shell python3.6 gives ImportError for lib while working when run from terminal.

ID:2130242
 
Aug 11 2016, 6:09 am
(See the best response by Nadrew.)
Code:
from urllib.request import build_opener
from bs4 import BeautifulSoup
from sys import argv

opener = build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')] #wikipedia needs this

url = str(argv[1])
#For testing from terminal use below and comment above.
#url = "https://apollo-community.org/wiki/index.php?title=Example_Paperwork"

resource = opener.open(url)
data = resource.read()
resource.close()
soup = BeautifulSoup(data, "html.parser")
data = soup.find('div',id="mw-content-text")

#Some preprocessing becuase byond is crap with html chars
data_string = str(data)
data_lines = list(data_string.splitlines(True))

#If pulled data is empty exit with 1 so the shell knows it failed
if len(data_lines) == 0:
    exit(1)
i = 0
while i < len(data_lines):
    if '<div class="mw-collapsible-content"><pre>' in data_lines[i]:
        data_lines[i] = "\n--start--\n"

    if "</pre></div></div>" in data_lines[i]:
        data_lines[i] = "\n--stop--"

    if "<" in data_lines[i] or ">" in data_lines[i]:
        data_lines[i] = ""

    i += 1

data_string = str(" ".join(str(x) for x in data_lines))
text_file = open("scripts/wikiForms.txt", "w", encoding='utf-8')
text_file.write(data_string)
text_file.close()
exit()


Problem description:
When I run this on my server (CentOS) with the command python3.6 script.py it runs just fine and gives disired output.
When I run it via:
shell("python3.6 script.py '[url]'"

It gives an error
ImportError: No module named request
I've made REALY sure request is installed on python3.6.
Anybody have any ideas ?
Maybe another lib I can use to pull html from a wiki page if this is a problem we cannot resolve ?
Aug 11 2016, 2:56 pm
Best response
You could use world.Export() to get the webpage content directly without having to use a middle-man.
Aug 11 2016, 4:02 pm
Nope you can't world export can only be used on non HTTPS pages. (ftp/http.. whaterver).
Aug 11 2016, 4:16 pm
Ah yeah, that'd put a damper on it.

As for the initial question, it's strange that it's showing request as failing but not urllib itself, seems to indicate that there's some kind of access problem. When you tested the script were you sure to try it using the same permissions the game would have?

You could try just importing urllib and calling the functions in long-form to see if it can access the request module that way, but I'd imagine you'd come across the same error. This edges into Python issues though, it's doubtful there's something on DM's end causing the issue, it could be something as simple as permission conflicts or something as complicated as an improperly installed module.
Aug 12 2016, 8:42 am
I've found the problem.
So our machine has python 2.6 and python 3.6 installed.
2.6 is the default so I use shell(python3.6 ..). However the dm shell environment only looks at the default python libs + any installed libs (aka BeautifulSoup). So it finds the python 2.6 urllib and the python 3.6 BeautifulSoup but not the python 3.6 urllib (wich contains request). To fix this you need to set the environmental variable PYTHONPATH (which serves as a secondary search location with the primary yielded no result) to the python3.6 location on your system. As far as I know this is only a problem on Linux and maybe even only on CentOS.