HPCE/ldapuserdir
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
ldapuserdir/todo.org
Derek Feichtinger 6da52b2950 Version for python 3.6 
- basic syntax fixes
- bytestring to utf8 conversions
- __init__.py import problem fix
2019-05-11 12:34:44 +02:00

472 lines
17 KiB
Org Mode
Raw Permalink Blame History

#+TITLE: todo for ldapuserdir
#+TODO: BUG(b!) FEATURE(f!) | RESOLVED(r!) WONTFIX(w!) WORKAROUND(W!)
#+DATE: <2013-07-22 Mon>
#+AUTHOR: Derek Feichtinger
#+EMAIL: derek.feichtinger@psi.ch
#+OPTIONS: ':nil *:t -:t ::t <:t H:3 \n:nil ^:t arch:headline
#+OPTIONS: author:t c:nil creator:comment d:(not LOGBOOK) date:t e:t
#+OPTIONS: email:nil f:t inline:t num:t p:nil pri:nil stat:t tags:t
#+OPTIONS: tasks:t tex:t timestamp:t toc:t todo:t |:t
#+CREATOR: Emacs 24.3.1 (Org mode 8.0.6)
#+DESCRIPTION:
#+EXCLUDE_TAGS: noexport
#+KEYWORDS:
#+LANGUAGE: en
#+SELECT_TAGS: export
* Bugs and feature requests
** RESOLVED Error when displayName not set
CLOSED: [2016-03-10 Thu 15:44]
:LOGBOOK:
- State "RESOLVED" from "BUG" [2016-03-10 Thu 15:44]
- State "BUG" from [2016-03-09 Wed 14:13]
:END:
#+BEGIN_EXAMPLE
dfeich@dflt1w:~/Documents/merlin-accounting$ ldapuserdir-ctl -n -u '*sz*'
monoszlai_b:36001:IGNORE:710:Monoszlai Balazs:/afs/psi.ch/user/m/monoszlai_b:/bin/bash
kupiszewski_p:4822:IGNORE:520:Kupiszewski Piotr:/afs/psi.ch/user/k/kupiszewski_p:/bin/bash
szijarto_r:3107:IGNORE:420:Szijarto Rita:/afs/psi.ch/user/s/szijarto_r:/bin/tcsh
janasz_f:4359:IGNORE:420:Janasz Filip:/afs/psi.ch/user/j/janasz_f:/bin/bash
Traceback (most recent call last):
File "/opt/anaconda/python2.7/bin/ldapuserdir-ctl", line 4, in <module>
__import__('pkg_resources').run_script('ldapuserdir==2.1.2', 'ldapuserdir-ctl')
File "/opt/anaconda/python2.7/lib/python2.7/site-packages/setuptools-20.1.1-py2.7.egg/pkg_resources/__init__.py", line 724, in run_script
File "/opt/anaconda/python2.7/lib/python2.7/site-packages/setuptools-20.1.1-py2.7.egg/pkg_resources/__init__.py", line 1657, in run_script
File "/opt/anaconda/python2.7/lib/python2.7/site-packages/ldapuserdir-2.1.2-py2.7.egg/EGG-INFO/scripts/ldapuserdir-ctl", line 345, in <module>
File "build/bdist.linux-x86_64/egg/ldapuserdir/ldapuserdir.py", line 359, in list_users_etcpwd
KeyError: 'displayName'
#+END_EXAMPLE
** RESOLVED Handling of ldap server returning slices (paging)
CLOSED: [2016-03-03 Thu 15:07]
:LOGBOOK:
- State "RESOLVED" from "BUG" [2016-03-03 Thu 15:07]
- State "BUG" from [2016-02-16 Tue 16:03]
:END:
- [ ] In the case of big groups, ldap returns the results with the attribute name
member being extended for the slice information. Need to implement the client
to be able to deal with this.
- Dani's docu may help https://intranet.psi.ch/AIT/AdLdapClientConfiguration
- Kai Kaminski supplied a fix in #c0fa2b0bbb7bfd58f4f5197936d1ca487fdb8c4d
** WORKAROUND group list returning some elements with dn=None
CLOSED: [2016-02-16 Tue 17:27]
:LOGBOOK:
- State "WORKAROUND" from "BUG" [2016-02-16 Tue 17:27]
- State "BUG" from [2016-02-16 Tue 16:03]
:END:
Invoking ldapuserdir-ctl in the following way leads to error
#+BEGIN_SRC
ldapuserdir-ctl --debug --group-ou='dc=d,dc=psi,dc=ch' 'p1299*'
#+END_SRC
#+BEGIN_EXAMPLE
p12990:IGNORE:12990:
p12991:IGNORE:12991:
p12992:IGNORE:12992:
p12993:IGNORE:12993:
p12994:IGNORE:12994:
p12995:IGNORE:12995:
p12996:IGNORE:12996:
p12997:IGNORE:12997:
p12998:IGNORE:12998:
p12999:IGNORE:12999:
Traceback (most recent call last):
File "/home/dfeich/conda-envs/build-test/bin/ldapuserdir-ctl", line 6, in <module>
exec(compile(open(__file__).read(), __file__, 'exec'))
File "/home/dfeich/projects/ldapuserdir/bin/ldapuserdir-ctl", line 341, in <module>
verbose=flag_verbose)
File "/home/dfeich/projects/ldapuserdir/ldapuserdir/ldapuserdir.py", line 537, in list_groups
sys.stdout.write("%s:IGNORE:%s:" % (entry['cn'][0], gid))
TypeError: list indices must be integers, not str
#+END_EXAMPLE
Reason: The search in get_groups_struct
#+BEGIN_SRC python
r = self.search_s_reconn(group_ou, ldap.SCOPE_SUBTREE, srch)
#+END_SRC
returns a result list which contains (dn, dict) pairs with the exception of the
last 2 entries which are (None, list).
#+BEGIN_EXAMPLE
[ ...
('CN=p12999,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch',
{'cn': ['p12999'],
'dSCorePropagationData': ['16010101000000.0Z'],
'distinguishedName': ['CN=p12999,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch'],
'gidNumber': ['12999'],
'groupType': ['-2147483646'],
'instanceType': ['4'],
'msSFU30GidNumber': ['12999'],
'name': ['p12999'],
'objectCategory': ['CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch'],
'objectClass': ['top', 'group'],
'objectGUID': ['jx\xd9\x7f/\x16\xebJ\xb7\x1c\xf3~\n\xd4\x17\x95'],
'objectSid': ['\x01\x05\x00\x00\x00\x00\x00\x05\x15\x00\x00\x00x.\x9d\x13\x931\xda\x1cC\x17\n2\xa9\x1a\x01\x00'],
'sAMAccountName': ['p12999'],
'sAMAccountType': ['268435456'],
'uSNChanged': ['817289847'],
'uSNCreated': ['817289847'],
'whenChanged': ['20160215133759.0Z'],
'whenCreated': ['20160215133757.0Z']}),
(None, ['ldaps://d.psi.ch/CN=Configuration,DC=d,DC=psi,DC=ch']),
(None, ['ldaps://d.psi.ch/CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch'])]
#+END_EXAMPLE
Not clear how these entries come to pass. It does not happen if I use the additional
classification 'ou=IT'.
#+BEGIN_SRC sh
ldapuserdir-ctl --debug --group-ou='ou=IT,dc=d,dc=psi,dc=ch' 'p1299*'
#+END_SRC
The corresponding ldapsearch yields no clue abou these additional elements. Probably
it filters them out. Discussing with Björn and looking it up in AD, we can see that
these CN paths exist for the config information.
#+BEGIN_SRC sh :results output
ldapsearch -x -w 'TBVsK5zOfqMyxVmXco7y' \
-D 'CN=linux_ldap,OU=Services,OU=IT,DC=d,DC=psi,DC=ch' \
-H "ldaps://d.psi.ch:636" -b "dc=d,dc=PSI,dc=ch" \
'(&(objectClass=Group)(msSFU30GidNumber=*)(cn=p1299*))'
#+END_SRC
#+RESULTS:
#+begin_example
# extended LDIF
#
# LDAPv3
# base <dc=d,dc=PSI,dc=ch> with scope subtree
# filter: (&(objectClass=Group)(msSFU30GidNumber=*)(cn=p1299*))
# requesting: ALL
#
# p12990, Groups, Experiment, IT, d.psi.ch
dn: CN=p12990,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch
objectClass: top
objectClass: group
cn: p12990
distinguishedName: CN=p12990,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch
instanceType: 4
whenCreated: 20160215133748.0Z
whenChanged: 20160215133757.0Z
uSNCreated: 817286825
uSNChanged: 817286825
name: p12990
objectGUID:: e0ffV/01TEOscX9xFanuew==
objectSid:: AQUAAAAAAAUVAAAAeC6dE5Mx2hxDFwoy+w4BAA==
sAMAccountName: p12990
sAMAccountType: 268435456
groupType: -2147483646
objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch
dSCorePropagationData: 16010101000000.0Z
msSFU30GidNumber: 12990
gidNumber: 12990
# p12991, Groups, Experiment, IT, d.psi.ch
dn: CN=p12991,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch
objectClass: top
objectClass: group
cn: p12991
distinguishedName: CN=p12991,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch
instanceType: 4
whenCreated: 20160215133751.0Z
whenChanged: 20160215133758.0Z
uSNCreated: 817287915
uSNChanged: 817287915
name: p12991
objectGUID:: 9+o2BvtkMkK3pDPbQrTyPw==
objectSid:: AQUAAAAAAAUVAAAAeC6dE5Mx2hxDFwoyLRMBAA==
sAMAccountName: p12991
sAMAccountType: 268435456
groupType: -2147483646
objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch
dSCorePropagationData: 16010101000000.0Z
msSFU30GidNumber: 12991
gidNumber: 12991
# p12992, Groups, Experiment, IT, d.psi.ch
dn: CN=p12992,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch
objectClass: top
objectClass: group
cn: p12992
distinguishedName: CN=p12992,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch
instanceType: 4
whenCreated: 20160215133755.0Z
whenChanged: 20160215133758.0Z
uSNCreated: 817289293
uSNChanged: 817289293
name: p12992
objectGUID:: llozR0I690u3dIbXoLFdpg==
objectSid:: AQUAAAAAAAUVAAAAeC6dE5Mx2hxDFwoyhBgBAA==
sAMAccountName: p12992
sAMAccountType: 268435456
groupType: -2147483646
objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch
dSCorePropagationData: 16010101000000.0Z
msSFU30GidNumber: 12992
gidNumber: 12992
# p12993, Groups, Experiment, IT, d.psi.ch
dn: CN=p12993,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch
objectClass: top
objectClass: group
cn: p12993
distinguishedName: CN=p12993,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch
instanceType: 4
whenCreated: 20160215133755.0Z
whenChanged: 20160215133758.0Z
uSNCreated: 817289294
uSNChanged: 817289294
name: p12993
objectGUID:: QY8deD1SsUaP+G9ShR3+5A==
objectSid:: AQUAAAAAAAUVAAAAeC6dE5Mx2hxDFwoyhRgBAA==
sAMAccountName: p12993
sAMAccountType: 268435456
groupType: -2147483646
objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch
dSCorePropagationData: 16010101000000.0Z
msSFU30GidNumber: 12993
gidNumber: 12993
# p12994, Groups, Experiment, IT, d.psi.ch
dn: CN=p12994,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch
objectClass: top
objectClass: group
cn: p12994
distinguishedName: CN=p12994,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch
instanceType: 4
whenCreated: 20160215133757.0Z
whenChanged: 20160215133759.0Z
uSNCreated: 817289842
uSNChanged: 817289842
name: p12994
objectGUID:: rsAueF9W2kaQ2VW/Qtp1wg==
objectSid:: AQUAAAAAAAUVAAAAeC6dE5Mx2hxDFwoypBoBAA==
sAMAccountName: p12994
sAMAccountType: 268435456
groupType: -2147483646
objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch
dSCorePropagationData: 16010101000000.0Z
msSFU30GidNumber: 12994
gidNumber: 12994
# p12995, Groups, Experiment, IT, d.psi.ch
dn: CN=p12995,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch
objectClass: top
objectClass: group
cn: p12995
distinguishedName: CN=p12995,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch
instanceType: 4
whenCreated: 20160215133757.0Z
whenChanged: 20160215133759.0Z
uSNCreated: 817289843
uSNChanged: 817289843
name: p12995
objectGUID:: xEDgBQTscEennB8f7HnrJg==
objectSid:: AQUAAAAAAAUVAAAAeC6dE5Mx2hxDFwoypRoBAA==
sAMAccountName: p12995
sAMAccountType: 268435456
groupType: -2147483646
objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch
dSCorePropagationData: 16010101000000.0Z
msSFU30GidNumber: 12995
gidNumber: 12995
# p12996, Groups, Experiment, IT, d.psi.ch
dn: CN=p12996,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch
objectClass: top
objectClass: group
cn: p12996
distinguishedName: CN=p12996,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch
instanceType: 4
whenCreated: 20160215133757.0Z
whenChanged: 20160215133759.0Z
uSNCreated: 817289844
uSNChanged: 817289844
name: p12996
objectGUID:: rzV1wOhoZ0GgI9B9Of+oFA==
objectSid:: AQUAAAAAAAUVAAAAeC6dE5Mx2hxDFwoyphoBAA==
sAMAccountName: p12996
sAMAccountType: 268435456
groupType: -2147483646
objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch
dSCorePropagationData: 16010101000000.0Z
msSFU30GidNumber: 12996
gidNumber: 12996
# p12997, Groups, Experiment, IT, d.psi.ch
dn: CN=p12997,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch
objectClass: top
objectClass: group
cn: p12997
distinguishedName: CN=p12997,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch
instanceType: 4
whenCreated: 20160215133757.0Z
whenChanged: 20160215133759.0Z
uSNCreated: 817289845
uSNChanged: 817289845
name: p12997
objectGUID:: 15jnQe7x7kubD0LVnwGhWA==
objectSid:: AQUAAAAAAAUVAAAAeC6dE5Mx2hxDFwoypxoBAA==
sAMAccountName: p12997
sAMAccountType: 268435456
groupType: -2147483646
objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch
dSCorePropagationData: 16010101000000.0Z
msSFU30GidNumber: 12997
gidNumber: 12997
# p12998, Groups, Experiment, IT, d.psi.ch
dn: CN=p12998,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch
objectClass: top
objectClass: group
cn: p12998
distinguishedName: CN=p12998,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch
instanceType: 4
whenCreated: 20160215133757.0Z
whenChanged: 20160215133759.0Z
uSNCreated: 817289846
uSNChanged: 817289846
name: p12998
objectGUID:: K0QCSvfKY0+8xBfUx/sp1A==
objectSid:: AQUAAAAAAAUVAAAAeC6dE5Mx2hxDFwoyqBoBAA==
sAMAccountName: p12998
sAMAccountType: 268435456
groupType: -2147483646
objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch
dSCorePropagationData: 16010101000000.0Z
msSFU30GidNumber: 12998
gidNumber: 12998
# p12999, Groups, Experiment, IT, d.psi.ch
dn: CN=p12999,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch
objectClass: top
objectClass: group
cn: p12999
distinguishedName: CN=p12999,OU=Groups,OU=Experiment,OU=IT,DC=d,DC=psi,DC=ch
instanceType: 4
whenCreated: 20160215133757.0Z
whenChanged: 20160215133759.0Z
uSNCreated: 817289847
uSNChanged: 817289847
name: p12999
objectGUID:: anjZfy8W60q3HPN+CtQXlQ==
objectSid:: AQUAAAAAAAUVAAAAeC6dE5Mx2hxDFwoyqRoBAA==
sAMAccountName: p12999
sAMAccountType: 268435456
groupType: -2147483646
objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=d,DC=psi,DC=ch
dSCorePropagationData: 16010101000000.0Z
msSFU30GidNumber: 12999
gidNumber: 12999
# search reference
ref: ldaps://d.psi.ch/CN=Configuration,DC=d,DC=psi,DC=ch
# search result
search: 2
result: 0 Success
# numResponses: 12
# numEntries: 10
# numReferences: 1
#+end_example
I implemented a workaround by filtering out the None elements.
** [2019-05-10 Fri] compatibility with python-3.6
*** RESOLVED simple fixes
CLOSED: [2019-05-11 Sat 09:54]
:LOGBOOK:
- State "RESOLVED" from "BUG" [2019-05-11 Sat 09:54]
- State "BUG" from [2019-05-11 Sat 09:54]
:END:
- Exceptions: use new syntax
#+begin_src python
except SomeException as err
#+end_src
- print statements
*** RESOLVED importer namespace problem
CLOSED: [2019-05-11 Sat 09:55]
:LOGBOOK:
- State "RESOLVED" from "BUG" [2019-05-11 Sat 09:55]
- State "BUG" from [2019-05-11 Sat 09:54]
:END:
- __init__.py only works with changing to relative import
: from ldapuserdir import LdapUserDir, LdapUserDirError
now must be made explicit with
: from ldapuserdir.ldapuserdir import LdapUserDir, LdapUserDirError
*** RESOLVED hangs in LDAP paging call
CLOSED: [2019-05-11 Sat 12:28]
:LOGBOOK:
- State "RESOLVED" from "BUG" [2019-05-11 Sat 12:28]
- State "BUG" from [2019-05-11 Sat 10:05]
:END:
The loop for reading the paged results never reaches the break condition
in ldapuserdir.py:_search_s
#+begin_src python
page_ctrl = SimplePagedResultsControl(criticality=True,
size=self.page_size,
cookie='')
msgid = self._ldap.search_ext(base, scope, filterstr, attrlist,
attrsonly,
serverctrls=[page_ctrl])
results = []
while True:
_, rdata, _, resp_ctrls = self._ldap.result3(msgid)
results.extend(rdata)
self.logger.debug('DEREK: in paging result call: results= %s' % results)
# .... CUT ....
# Extract the SimplePagedResultsControl to get the cookie.
page_ctrls = [c for c in resp_ctrls if c.controlType == SimplePagedResultsControl.controlType]
if page_ctrls == [] or page_ctrls[0].cookie == '':
# We're done.
break
else:
# Update the cookie to retrieve the next page.
page_ctrl.cookie = page_ctrls[0].cookie
#+end_src
The conditions for the break need to be changed.
Good resource: https://medium.com/@alpolishchuk/pagination-of-ldap-search-results-with-python-ldap-845de60b90d2
#+begin_src python
if not page_ctrls:
raise RuntimeError("The server ignores RFC 2696 control (paged results)")
if not page_ctrls[0].cookie:
# We're done.
break
# Update the cookie to retrieve the next page.
page_ctrl.cookie = page_ctrls[0].cookie
#+end_src
*** RESOLVED In python3 the ldap calls return bytestrings
CLOSED: [2019-05-11 Sat 12:28]
:LOGBOOK:
- State "RESOLVED" from "BUG" [2019-05-11 Sat 12:28]
- State "BUG" from [2019-05-11 Sat 12:28]
:END:
: ldapuserdir-ctl --debug -u feichtinger
: b'feichtinger':b'3896':b'3896':b'840':b'Feichtinger Derek Heinrich':b'/bin/bash':b'/afs/psi.ch/user/f/feichtinger':
The (dn, attributes) that are returned by _search_s contain attributes the
values of which all are bytestrings.
python-ldap returns bytestrings and in py3 a standard string is now utf-8.
This leads to all kinds of problems. I define a function
ensure_utf8 ẗo fix the issue.
Reference in New Issue View Git Blame Copy Permalink