Yrurtj

I ran this command on Ubuntu 14.04 Linux while trying to install a package:

export PYTHONPATH=/usr/local/lib/python2.7/ && sudo python setup.py install

The package I was trying to install is http://github.com/ewels/MultiQC

Then lots of things python related have gone wrong since this. For example, virtualenv doesn't work anymore, and this package doesn't work either:

http://wiki.dnanexus.com/images/files/dx-toolkit-v0.240.1-ubuntu-14.04-amd64.tar.gz

source Downloads/dx-toolkit/environment

dx login

Traceback (most recent call last):

  File "/home/avilella/Downloads/dx-toolkit/bin/dx", line 7, in <module>

    from dxpy.scripts.dx import main

  File "/home/avilella/Downloads/dx-toolkit/share/dnanexus/lib/python2.7/site-packages/dxpy/__init__.py", line 143, in <module>

    from . import exceptions

  File "/home/avilella/Downloads/dx-toolkit/share/dnanexus/lib/python2.7/site-packages/dxpy/exceptions.py", line 211, in <module>

    network_exceptions = (requests.packages.urllib3.exceptions.ProtocolError,

AttributeError: 'module' object has no attribute 'packages'

How do I revert this to the stage before running this command-line? Even starting from a fresh bash terminal, it now won't find any of my other python libraries.
Even with a freshly created Linux user on the same computer, the errors are the same.

Related question wasn't answered: Fixing a broken PYTHONPATH environment variable

[EDIT]

I don't know how to revert to the list of folders before, but I presume it was a combination of /usr/lib/python2.7 and /usr/local/lib/python2.7

Trying virtualenv also fails now:

virtualenv

Fatal Python error: Py_Initialize: Unable to get the locale encoding

  File "/usr/lib/python2.7/encodings/__init__.py", line 123

    raise CodecRegistryError,

                            ^

SyntaxError: invalid syntax

Aborted (core dumped)

Interactive python works to the extend I tried below:

avilella@ubuntu14:~$ python

Python 2.7.6 (default, Oct 26 2016, 20:30:19) 

[GCC 4.8.4] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> print "foo"

foo

>>> import urllib3

>>> 

I have now gone through the folder and tried both pip install and apt-get install python-${package} with this:

ls /usr/local/lib/python2.7/dist-packages/*/ | grep -v egg | grep ':' | tr ':' ' ' | grep -v dist-info | xargs -n1 basename  | while read i; do sudo pip install $i; done

Also:

ls /usr/local/lib/python2.7/dist-packages/*/ | grep -v egg | grep ':' | tr ':' ' ' | grep -v dist-info | xargs -n1 basename  | while read i; do sudo apt-get install -y python-${i}; done

But still same issues.

Package requests is there:

ls -l /usr/lib/python2.7/dist-packages/requests/

total 304

-rw-r--r-- 1 root root 14594 Mar 26  2015 adapters.py

-rw-rw-r-- 1 root root 13750 Sep  4 16:05 adapters.pyc

-rw-r--r-- 1 root root  4333 Sep 24  2013 api.py

-rw-rw-r-- 1 root root  5261 Sep  4 16:05 api.pyc

-rw-r--r-- 1 root root  6173 Dec  5  2013 auth.py

-rw-rw-r-- 1 root root  7206 Sep  4 16:05 auth.pyc

-rw-r--r-- 1 root root   527 Mar 26  2015 certs.py

-rw-rw-r-- 1 root root   736 Sep  4 16:05 certs.pyc

-rw-r--r-- 1 root root  2531 Mar 26  2015 compat.py

-rw-rw-r-- 1 root root  2541 Sep  4 16:05 compat.pyc

-rw-r--r-- 1 root root 16686 Jan  8  2014 cookies.py

-rw-rw-r-- 1 root root 20167 Sep  4 16:05 cookies.pyc

-rw-r--r-- 1 root root  1599 Mar 26  2015 exceptions.py

-rw-rw-r-- 1 root root  3863 Sep  4 16:05 exceptions.pyc

-rw-r--r-- 1 root root   820 Sep 24  2013 hooks.py

-rw-rw-r-- 1 root root  1055 Sep  4 16:05 hooks.pyc

-rw-r--r-- 1 root root  1846 Mar 26  2015 __init__.py

-rw-rw-r-- 1 root root  2501 Sep  4 16:05 __init__.pyc

-rw-r--r-- 1 root root 25332 Mar 26  2015 models.py

-rw-rw-r-- 1 root root 23051 Sep  4 16:05 models.pyc

-rw-r--r-- 1 root root 22042 Mar 26  2015 sessions.py

-rw-rw-r-- 1 root root 18049 Sep  4 16:05 sessions.pyc

-rw-r--r-- 1 root root  3136 Sep 24  2013 status_codes.py

-rw-rw-r-- 1 root root  4439 Sep  4 16:05 status_codes.pyc

-rw-r--r-- 1 root root  3575 Sep 24  2013 structures.py

-rw-rw-r-- 1 root root  6359 Sep  4 16:05 structures.pyc

-rw-r--r-- 1 root root 19642 Mar 26  2015 utils.py

-rw-rw-r-- 1 root root 19386 Sep  4 16:05 utils.pyc

I checked a working version of a different VM I have, where everything python related works, and the requests folder contains exactly the same files.

Any mentions to urllib in the non-working Linux (below) are the same as in the working one (further below), with 2 files changing their date between one an the other:

[       4096 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3

[       3346 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/_collections.pyc

[      22000 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/connectionpool.pyc

[       4096 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/contrib

[       6015 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/exceptions.pyc

[       6511 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/fields.pyc

[       3162 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/filepost.pyc

[       2074 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/__init__.pyc

[       4096 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/packages

[       9076 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/poolmanager.pyc

[       5850 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/request.pyc

[       9879 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/response.pyc

[      19662 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/util.pyc

[      28639 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py

[       5961 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/fields.py

[       2486 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/filepost.py

[      10159 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/response.py

[      20575 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/util.py

[       1703 Sep 25  2013]  /usr/lib/python2.7/dist-packages/urllib3/__init__.py

[       3274 Sep 25  2013]  /usr/lib/python2.7/dist-packages/urllib3/exceptions.py

[       2898 Aug 14  2013]  /usr/lib/python2.7/dist-packages/urllib3/_collections.py

[       9009 Aug 14  2013]  /usr/lib/python2.7/dist-packages/urllib3/poolmanager.py

[       5874 Aug 14  2013]  /usr/lib/python2.7/dist-packages/urllib3/request.py

[      17079 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3-1.7.1.egg-info

[       7142 Oct  5 14:47]  /usr/lib/python2.7/lib2to3/fixes/fix_urllib.pyc

[       8385 Oct 26  2016]  /usr/lib/python2.7/lib2to3/fixes/fix_urllib.py

[      46549 Oct  5 14:47]  /usr/lib/python2.7/urllib2.pyc

[      50626 Oct  5 14:47]  /usr/lib/python2.7/urllib.pyc

[      51788 Oct 26  2016]  /usr/lib/python2.7/urllib2.py

[      59371 Oct 26  2016]  /usr/lib/python2.7/urllib.py

The working one below:

[       4096 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3

[       3346 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/_collections.pyc

[      22000 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/connectionpool.pyc

[       4096 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/contrib

[       6015 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/exceptions.pyc

[       6511 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/fields.pyc

[       3162 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/filepost.pyc

[       2074 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/__init__.pyc

[       4096 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/packages

[       9076 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/poolmanager.pyc

[       5850 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/request.pyc

[       9879 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/response.pyc

[      19662 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/util.pyc

[      28639 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py

[       5961 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/fields.py

[       2486 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/filepost.py

[      10159 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/response.py

[      20575 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/util.py

[       1703 Sep 25  2013]  /usr/lib/python2.7/dist-packages/urllib3/__init__.py

[       3274 Sep 25  2013]  /usr/lib/python2.7/dist-packages/urllib3/exceptions.py

[       2898 Aug 14  2013]  /usr/lib/python2.7/dist-packages/urllib3/_collections.py

[       9009 Aug 14  2013]  /usr/lib/python2.7/dist-packages/urllib3/poolmanager.py

[       5874 Aug 14  2013]  /usr/lib/python2.7/dist-packages/urllib3/request.py

[      17079 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3-1.7.1.egg-info

[      46549 Jul 12 11:47]  /usr/lib/python2.7/urllib2.pyc

[      50626 Jul 12 11:47]  /usr/lib/python2.7/urllib.pyc

[       7142 Jul 12 11:47]  /usr/lib/python2.7/lib2to3/fixes/fix_urllib.pyc

[       8385 Oct 26  2016]  /usr/lib/python2.7/lib2to3/fixes/fix_urllib.py

[      51788 Oct 26  2016]  /usr/lib/python2.7/urllib2.py

[      59371 Oct 26  2016]  /usr/lib/python2.7/urllib.py

Another example tool that doesn't work anymore is tosheets https://github.com/kren1/tosheets:

tosheets

Traceback (most recent call last):

  File "/usr/local/bin/tosheets", line 7, in <module>

    from tosheets.tosheets import main

  File "/usr/local/lib/python2.7/dist-packages/tosheets/tosheets.py", line 27, in <module>

    from apiclient import discovery

ImportError: cannot import name discovery

After doing unset PYTHON, then:

>>> import sys

>>> sys.path

['', '/usr/local/lib/python2.7/dist-packages/multiqc-1.4dev-py2.7.egg', '/usr/lib/python2.7/dist-packages', '/usr/lib/pymodules/python2.7', '/home/avilella/Downloads/dx-toolkit/share/dnanexus/lib/python2.7/site-packages', '/home/avilella/Downloads/dx-toolkit/lib/python', '/home/avilella', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/avilella/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PILcompat', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']

unset PYTHON, the python3:

>>> import sys

>>> sys.path

['', '/home/avilella/Downloads/dx-toolkit/share/dnanexus/lib/python2.7/site-packages', '/home/avilella/Downloads/dx-toolkit/lib/python', '/home/avilella', '/usr/lib/python3.4', '/usr/lib/python3.4/plat-x86_64-linux-gnu', '/usr/lib/python3.4/lib-dynload', '/usr/local/lib/python3.4/dist-packages', '/usr/lib/python3/dist-packages']

The contents of $PYTHONPATH before doing any source'ing are empty.

The contents of Downloads/dx-toolkit/environment are below:

# -*- Mode: shell-script -*-                                                                                                                                                                                                                                                                                                                                                                                                                                                                   

#                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              

# Copyright (C) 2013-2016 DNAnexus, Inc.                                                                                                                                                                                                                                                                                                                                                                                                                                                       

#                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              

# This file is part of dx-toolkit (DNAnexus platform client libraries).                                                                                                                                                                                                                                                                                                                                                                                                                        

#                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              

#   Licensed under the Apache License, Version 2.0 (the "License"); you may not                                                                                                                                                                                                                                                                                                                                                                                                                

#   use this file except in compliance with the License. You may obtain a copy                                                                                                                                                                                                                                                                                                                                                                                                                 

#   of the License at                                                                                                                                                                                                                                                                                                                                                                                                                                                                          

#                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              

#       http://www.apache.org/licenses/LICENSE-2.0                                                                                                                                                                                                                                                                                                                                                                                                                                             

#                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              

#   Unless required by applicable law or agreed to in writing, software                                                                                                                                                                                                                                                                                                                                                                                                                        

#   distributed under the License is distributed on an "AS IS" BASIS, WITHOUT                                                                                                                                                                                                                                                                                                                                                                                                                  

#   WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the                                                                                                                                                                                                                                                                                                                                                                                                                   

#   License for the specific language governing permissions and limitations                                                                                                                                                                                                                                                                                                                                                                                                                    

#   under the License.                                                                                                                                                                                                                                                                                                                                                                                                                                                                         

#                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              

#                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              

# Source this file in a bash shell to initialize DNAnexus environment                                                                                                                                                                                                                                                                                                                                                                                                                          

# variables:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   

# $ source environment                                                                                                                                                                                                                                                                                                                                                                                                                                                                         



# This line is compatible with both bash and csh/tcsh. If running in csh, it sources the file "environment.csh" and quits.                                                                                                                                                                                                                                                                                                                                                                     



test ! -z "$version" && echo "$shell" | grep -q csh && set SOURCE=`echo $_ | cut -f 2 -d " "` && set SOURCE=`dirname "$SOURCE"` && source "$SOURCE/environment.csh" && exit



# Resolve the location of this file                                                                                                                                                                                                                                                                                                                                                                                                                                                            

SOURCE="${BASH_SOURCE[0]}"

while [ -h "$SOURCE" ] ; do SOURCE="$(readlink "$SOURCE")"; done



# Get RHEL version                                                                                                                                                                                                                                                                                                                                                                                                                                                                             

if [ -e /etc/redhat-release ] ; then

  RHEL_MAJOR_VERSION=$(grep -o "Red Hat Enterprise Linux .* release [0-9]+" /etc/redhat-release | sed -e "s/Red Hat Enterprise Linux .* release //")

fi



export DNANEXUS_HOME="$( cd -P "$( dirname "$SOURCE" )" && pwd )"



# Detect system installation of dx-toolkit                                                                                                                                                                                                                                                                                                                                                                                                                                                     

if [ "$DNANEXUS_HOME" == "/etc/profile.d" ]; then



  export DNANEXUS_HOME="/usr/share/dnanexus"

  # Private Python packages. We really ought not pollute PYTHONPATH with these though.                                                                                                                                                                                                                                                                                                                                                                                                         

  export PYTHONPATH="/usr/share/dnanexus/lib/python2.7/site-packages:$PYTHONPATH"

  export CLASSPATH="/usr/share/java/dnanexus-api-0.1.0.jar:${CLASSPATH}"



else



  export PATH="${DNANEXUS_HOME}/bin:$PATH"

  export CLASSPATH="${DNANEXUS_HOME}/lib/java/*:${CLASSPATH}"



  if [ "$RHEL_MAJOR_VERSION" == "7" ]; then

    export PYTHONPATH="${DNANEXUS_HOME}/share/dnanexus/lib/python2.7/site-packages:${DNANEXUS_HOME}/lib64/python2.7/site-packages:$PYTHONPATH"

  else

    export PYTHONPATH="${DNANEXUS_HOME}/share/dnanexus/lib/python2.7/site-packages:${DNANEXUS_HOME}/lib/python:$PYTHONPATH"

  fi



fi



# Note: The default I/O stream encoding in Python 2.7 (as configured on ubuntu) is ascii, not UTF-8 or the system locale                                                                                                                                                                                                                                                                                                                                                                       

# encoding. We reset it here to avoid having to set it for every I/O operation explicitly.                                                                                                                                                                                                                                                                                                                                                                                                     

export PYTHONIOENCODING=UTF-8



eval "$(register-python-argcomplete dx|sed 's/-o default//')"



# Clean up old session files                                                                                                                                                                                                                                                                                                                                                                                                                                                                   

(shopt -s nullglob;

 if [[ $HOME != "" ]]; then

     for session_dir in "${HOME}/.dnanexus_config/sessions/"*; do

         if ! ps -p $(basename "$session_dir") &> /dev/null; then

             rm -rf "$session_dir"

         fi

     done

 fi

)



if [[ -z "$DX_SECURITY_CONTEXT" && -n "$DX_AUTH_TOKEN" ]]; then

    export DX_SECURITY_CONTEXT="{"auth_token_type":"Bearer","auth_token":"$DX_AUTH_TOKEN"}"

fi

I have a similar Ubuntu 14.04 VM working, would it be possible to rsync certain folders to it so that the non-working Ubuntu 14.04 is back to normal?

I seem to be stuck at this point, not knowing how to proceed any further.

I found the only solution is to reinstall the OS from scratch.