#+TITLE: todo for ldapuserdir #+TODO: BUG(b!) FEATURE(f!) | RESOLVED(r!) WONTFIX(w!) WORKAROUND(W!) #+DATE: <2013-07-22 Mon> #+AUTHOR: Derek Feichtinger #+EMAIL: derek.feichtinger@psi.ch #+OPTIONS: ':nil *:t -:t ::t <:t H:3 \n:nil ^:t arch:headline #+OPTIONS: author:t c:nil creator:comment d:(not LOGBOOK) date:t e:t #+OPTIONS: email:nil f:t inline:t num:t p:nil pri:nil stat:t tags:t #+OPTIONS: tasks:t tex:t timestamp:t toc:t todo:t |:t #+CREATOR: Emacs 24.3.1 (Org mode 8.0.6) #+DESCRIPTION: #+EXCLUDE_TAGS: noexport #+KEYWORDS: #+LANGUAGE: en #+SELECT_TAGS: export * Bugs and feature requests ** RESOLVED Error when displayName not set CLOSED: [2016-03-10 Thu 15:44] :LOGBOOK: - State "RESOLVED" from "BUG" [2016-03-10 Thu 15:44] - State "BUG" from [2016-03-09 Wed 14:13] :END: #+BEGIN_EXAMPLE dfeich@dflt1w:~/Documents/merlin-accounting$ ldapuserdir-ctl -n -u '*sz*' monoszlai_b:36001:IGNORE:710:Monoszlai Balazs:/afs/psi.ch/user/m/monoszlai_b:/bin/bash kupiszewski_p:4822:IGNORE:520:Kupiszewski Piotr:/afs/psi.ch/user/k/kupiszewski_p:/bin/bash szijarto_r:3107:IGNORE:420:Szijarto Rita:/afs/psi.ch/user/s/szijarto_r:/bin/tcsh janasz_f:4359:IGNORE:420:Janasz Filip:/afs/psi.ch/user/j/janasz_f:/bin/bash Traceback (most recent call last): File "/opt/anaconda/python2.7/bin/ldapuserdir-ctl", line 4, in __import__('pkg_resources').run_script('ldapuserdir==2.1.2', 'ldapuserdir-ctl') File "/opt/anaconda/python2.7/lib/python2.7/site-packages/setuptools-20.1.1-py2.7.egg/pkg_resources/__init__.py", line 724, in run_script File "/opt/anaconda/python2.7/lib/python2.7/site-packages/setuptools-20.1.1-py2.7.egg/pkg_resources/__init__.py", line 1657, in run_script File "/opt/anaconda/python2.7/lib/python2.7/site-packages/ldapuserdir-2.1.2-py2.7.egg/EGG-INFO/scripts/ldapuserdir-ctl", line 345, in File "build/bdist.linux-x86_64/egg/ldapuserdir/ldapuserdir.py", line 359, in list_users_etcpwd KeyError: 'displayName' #+END_EXAMPLE ** RESOLVED Handling of ldap server returning slices (paging) CLOSED: [2016-03-03 Thu 15:07] :LOGBOOK: - State "RESOLVED" from "BUG" [2016-03-03 Thu 15:07] - State "BUG" from [2016-02-16 Tue 16:03] :END: - [ ] In the case of big groups, ldap returns the results with the attribute name member being extended for the slice information. Need to implement the client to be able to deal with this. - Dani's docu may help https://intranet.psi.ch/AIT/AdLdapClientConfiguration - Kai Kaminski supplied a fix in #c0fa2b0bbb7bfd58f4f5197936d1ca487fdb8c4d ** WORKAROUND group list returning some elements with dn=None CLOSED: [2016-02-16 Tue 17:27] :LOGBOOK: - State "WORKAROUND" from "BUG" [2016-02-16 Tue 17:27] - State "BUG" from [2016-02-16 Tue 16:03] :END: Invoking ldapuserdir-ctl in the following way leads to error #+BEGIN_SRC ldapuserdir-ctl --debug --group-ou='dc=d,dc=psi,dc=ch' 'p1299*' #+END_SRC #+BEGIN_EXAMPLE p12990:IGNORE:12990: p12991:IGNORE:12991: p12992:IGNORE:12992: p12993:IGNORE:12993: p12994:IGNORE:12994: p12995:IGNORE:12995: p12996:IGNORE:12996: p12997:IGNORE:12997: p12998:IGNORE:12998: p12999:IGNORE:12999: Traceback (most recent call last): File "/home/dfeich/conda-envs/build-test/bin/ldapuserdir-ctl", line 6, in exec(compile(open(__file__).read(), __file__, 'exec')) File "/home/dfeich/projects/ldapuserdir/bin/ldapuserdir-ctl", line 341, in verbose=flag_verbose) File "/home/dfeich/projects/ldapuserdir/ldapuserdir/ldapuserdir.py", line 537, in list_groups sys.stdout.write("%s:IGNORE:%s:" % (entry['cn'][0], gid)) TypeError: list indices must be integers, not str #+END_EXAMPLE Reason: The search in get_groups_struct #+BEGIN_SRC python r = self.search_s_reconn(group_ou, ldap.SCOPE_SUBTREE, srch) #+END_SRC returns a result list which contains (dn, dict) pairs with the exception of the last 2 entries which are (None, list). #+BEGIN_EXAMPLE [ ... ('CN=p12999,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch', {'cn': ['p12999'], 'dSCorePropagationData': ['16010101000000.0Z'], 'distinguishedName': ['CN=p12999,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch'], 'gidNumber': ['12999'], 'groupType': ['-2147483646'], 'instanceType': ['4'], 'msSFU30GidNumber': ['12999'], 'name': ['p12999'], 'objectCategory': ['CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch'], 'objectClass': ['top', 'group'], 'objectGUID': ['jx\xd9\x7f/\x16\xebJ\xb7\x1c\xf3~\n\xd4\x17\x95'], 'objectSid': ['\x01\x05\x00\x00\x00\x00\x00\x05\x15\x00\x00\x00x.\x9d\x13\x931\xda\x1cC\x17\n2\xa9\x1a\x01\x00'], 'sAMAccountName': ['p12999'], 'sAMAccountType': ['268435456'], 'uSNChanged': ['817289847'], 'uSNCreated': ['817289847'], 'whenChanged': ['20160215133759.0Z'], 'whenCreated': ['20160215133757.0Z']}), (None, ['ldaps://d.psi.ch/CN=Configuration,DC=d,DC=psi,DC=ch']), (None, ['ldaps://d.psi.ch/CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch'])] #+END_EXAMPLE Not clear how these entries come to pass. It does not happen if I use the additional classification 'ou=IT'. #+BEGIN_SRC sh ldapuserdir-ctl --debug --group-ou='ou=IT,dc=d,dc=psi,dc=ch' 'p1299*' #+END_SRC The corresponding ldapsearch yields no clue abou these additional elements. Probably it filters them out. Discussing with Björn and looking it up in AD, we can see that these CN paths exist for the config information. #+BEGIN_SRC sh :results output ldapsearch -x -w 'TBVsK5zOfqMyxVmXco7y' \ -D 'CN=linux_ldap,OU=Services,OU=IT,DC=d,DC=psi,DC=ch' \ -H "ldaps://d.psi.ch:636" -b "dc=d,dc=PSI,dc=ch" \ '(&(objectClass=Group)(msSFU30GidNumber=*)(cn=p1299*))' #+END_SRC #+RESULTS: #+begin_example # extended LDIF # # LDAPv3 # base with scope subtree # filter: (&(objectClass=Group)(msSFU30GidNumber=*)(cn=p1299*)) # requesting: ALL # # p12990, Groups, Experiment, IT, d.psi.ch dn: CN=p12990,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch objectClass: top objectClass: group cn: p12990 distinguishedName: CN=p12990,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch instanceType: 4 whenCreated: 20160215133748.0Z whenChanged: 20160215133757.0Z uSNCreated: 817286825 uSNChanged: 817286825 name: p12990 objectGUID:: e0ffV/01TEOscX9xFanuew== objectSid:: AQUAAAAAAAUVAAAAeC6dE5Mx2hxDFwoy+w4BAA== sAMAccountName: p12990 sAMAccountType: 268435456 groupType: -2147483646 objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch dSCorePropagationData: 16010101000000.0Z msSFU30GidNumber: 12990 gidNumber: 12990 # p12991, Groups, Experiment, IT, d.psi.ch dn: CN=p12991,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch objectClass: top objectClass: group cn: p12991 distinguishedName: CN=p12991,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch instanceType: 4 whenCreated: 20160215133751.0Z whenChanged: 20160215133758.0Z uSNCreated: 817287915 uSNChanged: 817287915 name: p12991 objectGUID:: 9+o2BvtkMkK3pDPbQrTyPw== objectSid:: AQUAAAAAAAUVAAAAeC6dE5Mx2hxDFwoyLRMBAA== sAMAccountName: p12991 sAMAccountType: 268435456 groupType: -2147483646 objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch dSCorePropagationData: 16010101000000.0Z msSFU30GidNumber: 12991 gidNumber: 12991 # p12992, Groups, Experiment, IT, d.psi.ch dn: CN=p12992,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch objectClass: top objectClass: group cn: p12992 distinguishedName: CN=p12992,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch instanceType: 4 whenCreated: 20160215133755.0Z whenChanged: 20160215133758.0Z uSNCreated: 817289293 uSNChanged: 817289293 name: p12992 objectGUID:: llozR0I690u3dIbXoLFdpg== objectSid:: AQUAAAAAAAUVAAAAeC6dE5Mx2hxDFwoyhBgBAA== sAMAccountName: p12992 sAMAccountType: 268435456 groupType: -2147483646 objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch dSCorePropagationData: 16010101000000.0Z msSFU30GidNumber: 12992 gidNumber: 12992 # p12993, Groups, Experiment, IT, d.psi.ch dn: CN=p12993,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch objectClass: top objectClass: group cn: p12993 distinguishedName: CN=p12993,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch instanceType: 4 whenCreated: 20160215133755.0Z whenChanged: 20160215133758.0Z uSNCreated: 817289294 uSNChanged: 817289294 name: p12993 objectGUID:: QY8deD1SsUaP+G9ShR3+5A== objectSid:: AQUAAAAAAAUVAAAAeC6dE5Mx2hxDFwoyhRgBAA== sAMAccountName: p12993 sAMAccountType: 268435456 groupType: -2147483646 objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch dSCorePropagationData: 16010101000000.0Z msSFU30GidNumber: 12993 gidNumber: 12993 # p12994, Groups, Experiment, IT, d.psi.ch dn: CN=p12994,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch objectClass: top objectClass: group cn: p12994 distinguishedName: CN=p12994,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch instanceType: 4 whenCreated: 20160215133757.0Z whenChanged: 20160215133759.0Z uSNCreated: 817289842 uSNChanged: 817289842 name: p12994 objectGUID:: rsAueF9W2kaQ2VW/Qtp1wg== objectSid:: AQUAAAAAAAUVAAAAeC6dE5Mx2hxDFwoypBoBAA== sAMAccountName: p12994 sAMAccountType: 268435456 groupType: -2147483646 objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch dSCorePropagationData: 16010101000000.0Z msSFU30GidNumber: 12994 gidNumber: 12994 # p12995, Groups, Experiment, IT, d.psi.ch dn: CN=p12995,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch objectClass: top objectClass: group cn: p12995 distinguishedName: CN=p12995,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch instanceType: 4 whenCreated: 20160215133757.0Z whenChanged: 20160215133759.0Z uSNCreated: 817289843 uSNChanged: 817289843 name: p12995 objectGUID:: xEDgBQTscEennB8f7HnrJg== objectSid:: AQUAAAAAAAUVAAAAeC6dE5Mx2hxDFwoypRoBAA== sAMAccountName: p12995 sAMAccountType: 268435456 groupType: -2147483646 objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch dSCorePropagationData: 16010101000000.0Z msSFU30GidNumber: 12995 gidNumber: 12995 # p12996, Groups, Experiment, IT, d.psi.ch dn: CN=p12996,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch objectClass: top objectClass: group cn: p12996 distinguishedName: CN=p12996,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch instanceType: 4 whenCreated: 20160215133757.0Z whenChanged: 20160215133759.0Z uSNCreated: 817289844 uSNChanged: 817289844 name: p12996 objectGUID:: rzV1wOhoZ0GgI9B9Of+oFA== objectSid:: AQUAAAAAAAUVAAAAeC6dE5Mx2hxDFwoyphoBAA== sAMAccountName: p12996 sAMAccountType: 268435456 groupType: -2147483646 objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch dSCorePropagationData: 16010101000000.0Z msSFU30GidNumber: 12996 gidNumber: 12996 # p12997, Groups, Experiment, IT, d.psi.ch dn: CN=p12997,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch objectClass: top objectClass: group cn: p12997 distinguishedName: CN=p12997,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch instanceType: 4 whenCreated: 20160215133757.0Z whenChanged: 20160215133759.0Z uSNCreated: 817289845 uSNChanged: 817289845 name: p12997 objectGUID:: 15jnQe7x7kubD0LVnwGhWA== objectSid:: AQUAAAAAAAUVAAAAeC6dE5Mx2hxDFwoypxoBAA== sAMAccountName: p12997 sAMAccountType: 268435456 groupType: -2147483646 objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch dSCorePropagationData: 16010101000000.0Z msSFU30GidNumber: 12997 gidNumber: 12997 # p12998, Groups, Experiment, IT, d.psi.ch dn: CN=p12998,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch objectClass: top objectClass: group cn: p12998 distinguishedName: CN=p12998,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch instanceType: 4 whenCreated: 20160215133757.0Z whenChanged: 20160215133759.0Z uSNCreated: 817289846 uSNChanged: 817289846 name: p12998 objectGUID:: K0QCSvfKY0+8xBfUx/sp1A== objectSid:: AQUAAAAAAAUVAAAAeC6dE5Mx2hxDFwoyqBoBAA== sAMAccountName: p12998 sAMAccountType: 268435456 groupType: -2147483646 objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch dSCorePropagationData: 16010101000000.0Z msSFU30GidNumber: 12998 gidNumber: 12998 # p12999, Groups, Experiment, IT, d.psi.ch dn: CN=p12999,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch objectClass: top objectClass: group cn: p12999 distinguishedName: CN=p12999,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch instanceType: 4 whenCreated: 20160215133757.0Z whenChanged: 20160215133759.0Z uSNCreated: 817289847 uSNChanged: 817289847 name: p12999 objectGUID:: anjZfy8W60q3HPN+CtQXlQ== objectSid:: AQUAAAAAAAUVAAAAeC6dE5Mx2hxDFwoyqRoBAA== sAMAccountName: p12999 sAMAccountType: 268435456 groupType: -2147483646 objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch dSCorePropagationData: 16010101000000.0Z msSFU30GidNumber: 12999 gidNumber: 12999 # search reference ref: ldaps://d.psi.ch/CN=Configuration,DC=d,DC=psi,DC=ch # search result search: 2 result: 0 Success # numResponses: 12 # numEntries: 10 # numReferences: 1 #+end_example I implemented a workaround by filtering out the None elements. ** [2019-05-10 Fri] compatibility with python-3.6 *** RESOLVED simple fixes CLOSED: [2019-05-11 Sat 09:54] :LOGBOOK: - State "RESOLVED" from "BUG" [2019-05-11 Sat 09:54] - State "BUG" from [2019-05-11 Sat 09:54] :END: - Exceptions: use new syntax #+begin_src python except SomeException as err #+end_src - print statements *** RESOLVED importer namespace problem CLOSED: [2019-05-11 Sat 09:55] :LOGBOOK: - State "RESOLVED" from "BUG" [2019-05-11 Sat 09:55] - State "BUG" from [2019-05-11 Sat 09:54] :END: - __init__.py only works with changing to relative import : from ldapuserdir import LdapUserDir, LdapUserDirError now must be made explicit with : from ldapuserdir.ldapuserdir import LdapUserDir, LdapUserDirError *** RESOLVED hangs in LDAP paging call CLOSED: [2019-05-11 Sat 12:28] :LOGBOOK: - State "RESOLVED" from "BUG" [2019-05-11 Sat 12:28] - State "BUG" from [2019-05-11 Sat 10:05] :END: The loop for reading the paged results never reaches the break condition in ldapuserdir.py:_search_s #+begin_src python page_ctrl = SimplePagedResultsControl(criticality=True, size=self.page_size, cookie='') msgid = self._ldap.search_ext(base, scope, filterstr, attrlist, attrsonly, serverctrls=[page_ctrl]) results = [] while True: _, rdata, _, resp_ctrls = self._ldap.result3(msgid) results.extend(rdata) self.logger.debug('DEREK: in paging result call: results= %s' % results) # .... CUT .... # Extract the SimplePagedResultsControl to get the cookie. page_ctrls = [c for c in resp_ctrls if c.controlType == SimplePagedResultsControl.controlType] if page_ctrls == [] or page_ctrls[0].cookie == '': # We're done. break else: # Update the cookie to retrieve the next page. page_ctrl.cookie = page_ctrls[0].cookie #+end_src The conditions for the break need to be changed. Good resource: https://medium.com/@alpolishchuk/pagination-of-ldap-search-results-with-python-ldap-845de60b90d2 #+begin_src python if not page_ctrls: raise RuntimeError("The server ignores RFC 2696 control (paged results)") if not page_ctrls[0].cookie: # We're done. break # Update the cookie to retrieve the next page. page_ctrl.cookie = page_ctrls[0].cookie #+end_src *** RESOLVED In python3 the ldap calls return bytestrings CLOSED: [2019-05-11 Sat 12:28] :LOGBOOK: - State "RESOLVED" from "BUG" [2019-05-11 Sat 12:28] - State "BUG" from [2019-05-11 Sat 12:28] :END: : ldapuserdir-ctl --debug -u feichtinger : b'feichtinger':b'3896':b'3896':b'840':b'Feichtinger Derek Heinrich':b'/bin/bash':b'/afs/psi.ch/user/f/feichtinger': The (dn, attributes) that are returned by _search_s contain attributes the values of which all are bytestrings. python-ldap returns bytestrings and in py3 a standard string is now utf-8. This leads to all kinds of problems. I define a function ensure_utf8 ẗo fix the issue.