Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transform.pd alignment issues #200

Closed
iainlane opened this issue Oct 31, 2016 · 8 comments
Closed

Transform.pd alignment issues #200

iainlane opened this issue Oct 31, 2016 · 8 comments

Comments

@iainlane
Copy link

(gdb) run
Starting program: /usr/bin/perl -Mblib t/transform.t
Cannot parse expression `.L954 4@r4'.
warning: Probes-based dynamic linker interface failed.
Reverting to original interface.

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
1..27
ok 1 - t_linear can make a 1-d transform
ok 2 - 1-d apply on a collection of vectors ignors higher dim
ok 3 - t_linear can make a 2-d transform
ok 4 - 2-d apply treats the higher dim
ok 5 - invert works
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12 - Bad values happen

Program received signal SIGBUS, Bus error.
0xf6cf159c in pdl_map_readdata (__tr=0xb3e1a8) at Transform.xs:5307
5307           index_stash[i] = i_off;
(gdb) bt
#0  0xf6cf159c in pdl_map_readdata (__tr=0xb3e1a8) at Transform.xs:5307
#1  0xf72f72e6 in pdl__ensure_trans (trans=0xb3e1a8, what=4) at pdlapi.c:1227
#2  0xf72f6734 in pdl_make_trans_mutual (trans=0xb3e1a8) at pdlapi.c:849
#3  0xf6cf3a44 in XS_PDL__map_int (my_perl=0x15e008, cv=0x789dc0) at Transform.xs:5728
#4  0x00099e46 in Perl_pp_entersub ()
#5  0x000950f4 in Perl_runops_standard ()
#6  0x000420c8 in perl_run ()
#7  0x000284a0 in main ()

As far as I can tell it's because this code doesn't ensure that the pointers are aligned. In Ubuntu we have kernels which raise SIGBUS on unaligned access, exposing this failure.

I've got a crude fix to align everything to 16 in this function; just testing it.

iainlane pushed a commit to iainlane/pdl that referenced this issue Oct 31, 2016
'ovec' is a contiguous collection of many different pointers. Each of
these may not be aligned, depending on the sizes that came before. We
round each sub-alignment to 16 bytes, so that the resulting pointers are
always on 16 byte boundaries.

On certain configurations (e.g. Ubuntu's armhf builders), unaligned
access raises a SIGBUS which terminates the program, so this is a real
problem.

Fixes PDLPorters#200.
@iainlane
Copy link
Author

See the referenced commit - I'm not at all sure it's right (new to alignment matters), so not proposing a PR. Feel free to cherry-pick if you want, or just use it for inspiration.

@devel-chm
Copy link
Member

Thanks for the report. Could you attach the output of perl -V to this ticket. Presumably, there should be something in the build options about making pointer 16byte aligned....I hope.

@iainlane
Copy link
Author

iainlane commented Nov 1, 2016

Ahh, I didn't know about perl -V

root@logical-moccasin:~# perl -V
Summary of my perl5 (revision 5 version 24 subversion 1) configuration:

  Platform:
    osname=linux, osvers=3.16.0, archname=arm-linux-gnueabihf-thread-multi-64int
    uname='linux localhost 3.16.0 #1 smp debian 3.16.0 armv7l gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dcc=arm-linux-gnueabihf-gcc -Dcpp=arm-linux-gnueabihf-cpp -Dld=arm-linux-gnueabihf-gcc -Dccflags=-DDEBIAN -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/build/perl-C8cTGe/perl-5.24.1~rc3=. -fstack-protector-strong -Wformat -Werror=format-security -Dldflags= -Wl,-Bsymbolic-functions -Wl,-z,relro -Dlddlflags=-shared -Wl,-Bsymbolic-functions -Wl,-z,relro -Dcccdlflags=-fPIC -Darchname=arm-linux-gnueabihf -Dprefix=/usr -Dprivlib=/usr/share/perl/5.24 -Darchlib=/usr/lib/arm-linux-gnueabihf/perl/5.24 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/arm-linux-gnueabihf/perl5/5.24 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.24.1 -Dsitearch=/usr/local/lib/arm-linux-gnueabihf/perl/5.24.1 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dusesitecustomize -Duse64bitint -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -Ui_libutil -Uversiononly -DDEBUGGING=-g -Doptimize=-O2 -dEs -Uuseshrplib -Dotherlibdirs=/usr/lib/arm-linux-gnueabihf/perl-base'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    use64bitint=define, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='arm-linux-gnueabihf-gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2 -g',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='6.2.0 20161019', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678, doublekind=3
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8, longdblkind=0
    ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='arm-linux-gnueabihf-gcc', ldflags =' -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib/gcc/arm-linux-gnueabihf/6/include-fixed /usr/include/arm-linux-gnueabihf /usr/lib /lib/arm-linux-gnueabihf /lib/../lib /usr/lib/arm-linux-gnueabihf /usr/lib/../lib /lib
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=libc-2.24.so, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.24'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib -fstack-protector-strong'


Characteristics of this binary (from libperl): 
  Compile-time options: HAS_TIMES MULTIPLICITY PERLIO_LAYERS
                        PERL_COPY_ON_WRITE PERL_DONT_CREATE_GVSV
                        PERL_HASH_FUNC_ONE_AT_A_TIME_HARD
                        PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP
                        PERL_PRESERVE_IVUV USE_64_BIT_INT USE_ITHREADS
                        USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE
                        USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME
                        USE_PERLIO USE_PERL_ATOF USE_REENTRANT_API
                        USE_SITECUSTOMIZE
  Locally applied patches:
    RC3
    DEBPKG:debian/cpan_definstalldirs - Provide a sensible INSTALLDIRS default for modules installed from CPAN.
    DEBPKG:debian/db_file_ver - http://bugs.debian.org/340047 Remove overly restrictive DB_File version check.
    DEBPKG:debian/doc_info - Replace generic man(1) instructions with Debian-specific information.
    DEBPKG:debian/enc2xs_inc - http://bugs.debian.org/290336 Tweak enc2xs to follow symlinks and ignore missing @INC directories.
    DEBPKG:debian/errno_ver - http://bugs.debian.org/343351 Remove Errno version check due to upgrade problems with long-running processes.
    DEBPKG:debian/libperl_embed_doc - http://bugs.debian.org/186778 Note that libperl-dev package is required for embedded linking
    DEBPKG:fixes/respect_umask - Respect umask during installation
    DEBPKG:debian/writable_site_dirs - Set umask approproately for site install directories
    DEBPKG:debian/extutils_set_libperl_path - EU:MM: set location of libperl.a under /usr/lib
    DEBPKG:debian/no_packlist_perllocal - Don't install .packlist or perllocal.pod for perl or vendor
    DEBPKG:debian/fakeroot - Postpone LD_LIBRARY_PATH evaluation to the binary targets.
    DEBPKG:debian/instmodsh_doc - Debian policy doesn't install .packlist files for core or vendor.
    DEBPKG:debian/ld_run_path - Remove standard libs from LD_RUN_PATH as per Debian policy.
    DEBPKG:debian/libnet_config_path - Set location of libnet.cfg to /etc/perl/Net as /usr may not be writable.
    DEBPKG:debian/mod_paths - Tweak @INC ordering for Debian
    DEBPKG:debian/prune_libs - http://bugs.debian.org/128355 Prune the list of libraries wanted to what we actually need.
    DEBPKG:fixes/net_smtp_docs - [rt.cpan.org #36038] http://bugs.debian.org/100195 Document the Net::SMTP 'Port' option
    DEBPKG:debian/perlivp - http://bugs.debian.org/510895 Make perlivp skip include directories in /usr/local
    DEBPKG:debian/deprecate-with-apt - http://bugs.debian.org/747628 Point users to Debian packages of deprecated core modules
    DEBPKG:debian/squelch-locale-warnings - http://bugs.debian.org/508764 Squelch locale warnings in Debian package maintainer scripts
    DEBPKG:debian/skip-upstream-git-tests - Skip tests specific to the upstream Git repository
    DEBPKG:debian/patchlevel - http://bugs.debian.org/567489 List packaged patches for 5.24.1~rc3-3 in patchlevel.h
    DEBPKG:debian/skip-kfreebsd-crash - http://bugs.debian.org/628493 [perl #96272] Skip a crashing test case in t/op/threads.t on GNU/kFreeBSD
    DEBPKG:fixes/document_makemaker_ccflags - http://bugs.debian.org/628522 [rt.cpan.org #68613] Document that CCFLAGS should include $Config{ccflags}
    DEBPKG:debian/find_html2text - http://bugs.debian.org/640479 Configure CPAN::Distribution with correct name of html2text
    DEBPKG:debian/perl5db-x-terminal-emulator.patch - http://bugs.debian.org/668490 Invoke x-terminal-emulator rather than xterm in perl5db.pl
    DEBPKG:debian/cpan-missing-site-dirs - http://bugs.debian.org/688842 Fix CPAN::FirstTime defaults with nonexisting site dirs if a parent is writable
    DEBPKG:fixes/memoize_storable_nstore - [rt.cpan.org #77790] http://bugs.debian.org/587650 Memoize::Storable: respect 'nstore' option not respected
    DEBPKG:debian/regen-skip - Skip a regeneration check in unrelated git repositories
    DEBPKG:debian/makemaker-pasthru - http://bugs.debian.org/758471 Pass LD settings through to subdirectories
    DEBPKG:debian/makemaker-manext - http://bugs.debian.org/247370 Make EU::MakeMaker honour MANnEXT settings in generated manpage headers
    DEBPKG:debian/devel-ppport-reproducibility - http://bugs.debian.org/801523 Sort the list of XS code files when generating RealPPPort.xs
    DEBPKG:debian/encode-unicode-bom-doc - http://bugs.debian.org/798727 Document Debian backport of Encode::Unicode fix
    DEBPKG:debian/kfreebsd-softupdates - http://bugs.debian.org/796798 Work around Debian Bug#796798
    DEBPKG:fixes/autodie-scope - http://bugs.debian.org/798096 Fix a scoping issue with "no autodie" and the "system" sub
    DEBPKG:fixes/crosscompile-no-targethost - [23695c0] [perl #127234] Fix the Configure escape with usecrosscompile but no targethost
    DEBPKG:fixes/memoize-pod - [rt.cpan.org #89441] Fix POD errors in Memoize
    DEBPKG:fixes/ok-pod - Added encoding for pod.
    DEBPKG:debian/hurd-softupdates - http://bugs.debian.org/822735 Fix t/op/stat.t failures on hurd
    DEBPKG:fixes/nntp_docs - http://bugs.debian.org/51962 Net::NNTP: Correct innd/nnrpd confusion in relation to Reader option
    DEBPKG:fixes/math_complex_doc_great_circle - http://bugs.debian.org/697567 [rt.cpan.org #114104] Math::Trig: clarify definition of great_circle_midpoint
    DEBPKG:fixes/math_complex_doc_see_also - http://bugs.debian.org/697568 [rt.cpan.org #114105] Math::Trig: add missing SEE ALSO
    DEBPKG:fixes/math_complex_doc_angle_units - http://bugs.debian.org/731505 [rt.cpan.org #114106] Math::Trig: document angle units
    DEBPKG:fixes/cpan_web_link - http://bugs.debian.org/367291 CPAN: Add link to main CPAN web site
    DEBPKG:fixes/time_piece_doc - http://bugs.debian.org/817925 Time::Piece: Improve documentation for add_months and add_years
    DEBPKG:fixes/perlbug-refactor - http://bugs.debian.org/822463 [perl #128020] perlbug: Refactor duplicated file reading code
    DEBPKG:fixes/perlbug-linewrap - http://bugs.debian.org/822463 [perl #128020] perlbug: wrap overly long lines
    DEBPKG:fixes/hurd_sigaction - http://bugs.debian.org/825016 [d54f4ed] ext/POSIX/t/sigaction.t: Skip uid and pid tests on GNU/Hurd
    DEBPKG:fixes/hurd_hints - [4694301] http://bugs.debian.org/825020 [perl #128279] Modify hints for Hurd per Debian ticket 825020.
    DEBPKG:fixes/extutils-parsexs-reproducibility - [perl #128517] http://bugs.debian.org/829296 Make the output of ExtUtils::ParseXS reproducible
    DEBPKG:debian/CVE-2016-1238/sitecustomize-in-etc - Look for sitecustomize.pl in /etc/perl rather than sitelib on Debian systems
    DEBPKG:debian/CVE-2016-1238/test-suite-without-dot - [perl #127810] Patch unit tests to explicitly insert "." into @INC when needed.
    DEBPKG:debian/CVE-2016-1238/eumm-without-dot - [perl #127810] Add PERL_USE_UNSAFE_INC support to EU::MM for fortify_inc support.
    DEBPKG:debian/CVE-2016-1238/cpan-without-dot - [perl #127810] Set PERL_USE_UNSAFE_INC for cpan usage
    DEBPKG:debian/document_inc_removal - Document in perlvar that we remove '.' from @INC by default
    DEBPKG:fixes/extutils_makemaker_reproducible - http://bugs.debian.org/835815 http://bugs.debian.org/834190 Make perllocal.pod files reproducible
    DEBPKG:debian/CVE-2016-1238/remove-inc-test - Remove test for '.' in @INC as it might not be
    DEBPKG:debian/customized - Update customized.dat for files patched in Debian
    DEBPKG:fixes/file_path_hurd_errno - File-Path: Fix test failure in Hurd due to hard-coded ENOENT
  Built under linux
  Compiled at Sep 25 2016 20:22:41
  @INC:
    /etc/perl
    /usr/local/lib/arm-linux-gnueabihf/perl/5.24.1
    /usr/local/share/perl/5.24.1
    /usr/lib/arm-linux-gnueabihf/perl5/5.24
    /usr/share/perl5
    /usr/lib/arm-linux-gnueabihf/perl/5.24
    /usr/share/perl/5.24
    /usr/local/lib/site_perl
    /usr/lib/arm-linux-gnueabihf/perl-base
    .

So that's actually 8-byte aligned if alignbytes=8 is right - the patch should hopefully be easy to adapt for that.

@devel-chm
Copy link
Member

For reference, here is a link with discussion of the issue(s):

Pointer Casting and Alignment

@devel-chm
Copy link
Member

Looking at the code in transform.pd, it seems the general fix needs to be to add proper consideration of memory alignment constraints during data allocation. From your perl -V output, you have ptrsize=4, doublesize=8, ivsize=sizeof(PDL_Index)=8, and alignbytes=8. A quick fix for this plaform would be to use some type of memalign to allocate the memory.

@devel-chm
Copy link
Member

It looks like the problem with the existing code is the generation of the pointers to the memory regions. It gets off for the char data where the calculation does not include the needed alignment. It looks like the calculations do not include little-endian versus big-endian hardware either.

@mohawk2
Copy link
Member

mohawk2 commented Dec 26, 2021

@iainlane I recently updated all of that bit of transform.pd to use C99 variable-length arrays rather than the malloc scheme that was causing these problems. (https://github.com/PDLPorters/pdl/blob/master/Libtmp/Transform/transform.pd#L635 if you're interested)

Could you possibly try the latest master version of PDL (or indeed the latest CPAN release, since it was part of 2.058) and see if you can repro the problem?

@mohawk2
Copy link
Member

mohawk2 commented Dec 26, 2021

In fact, I will be bold and close this on the assumption that it's fixed, but please reopen and/or comment if that's not the case!

@mohawk2 mohawk2 closed this as completed Dec 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants