Hi! When implementing a python web scraper using scrapy for a Flask web app, I r

Question

Hi! When implementing a python web scraper using scrapy for a Flask web app, I ran into trouble trying to pass a variable between files. I declare a variable in a file called application.py, and then I need to use that variable inside of the start_url in my spider inside of another file. Part of the problem is that the file that I have to pass it to is located down two subdirectories (i.e. my web app lives in a project folder, application.py is in that folder, and the file I need to access is inside of a folder that itself is inside of a folder and that project directory). So, the structure is like this: my project folder contains application.py and folder a. Folder a contains folder b. and Folder b contains the file that I have to pass the variable to. I have tried things such as global variables, using a third file and importing to both application.py and the other file, and also I tried navigating the subdirectories, all of which failed. I think another issue here is that the nature of the spider somehow inhibits me from importing things normally.

Any suggestions as to how I could proceed with this?

Explanation / Answer

I can give you some ways to go to that subdirectory path :

import os

def readFile(filename):
   filehandle = open(filename)
   print filehandle.read()
   filehandle.close()

fileDir = os.path.dirname(os.path.relapath('_file_'))
print fileDir

#For accessing the file in the same folder
filename = "sample.txt"
readfile(filename)

#for accessing the file in a folder contained in the current folder
filename = os.path.join(fileDir, 'Folder1.1/sample.txt' )
readfile(filename)

#for accessing the file in the parent folder of the current folder
filename = os.path.join(fileDir, '../sample.txt' )
readfile(filename)

#for accessing the file inside a sibling folder
filename = os.path.join(fileDir, '../Folder2/sample.txt' )
filename = os.path.abspath(os.path.relapath(filename))
print filename
readfile(filename)

Another method :

if you have folder hierrarchy like this :
YourFolder/application.py
YourFolder/AnotherFolder/OneScript.py

than,you can use os module like this :

import sys
import os

def call_func(func,*args,**opts) :
   prev = os.path.abspath(os.getcwd()) #save the real cwd
try :
   #get the child's physical location.
   func_path = sys.modules[func._module_]._file_
execpt :
   #Die, we got passed a built-in function or you could just call it return func(*args, **opts)
return none
os.chdir(func_path)    #change to expected directory and run the fucntion
result = func(*args,**opts)

#fix the cwd and return
os.chdir(prev)
return result.

import ChildOneScript.ChildOne as ChildOne
call_func(ChildOne.myChildfunction, 0 ,1, kwaeg =false)

===================================
Hope this helps.

Related Questions

Navigate

• Browse (All)
• Browse H
• Subjects

Integrity-first tutoring: explanations and feedback only — we do not complete graded work. Learn more.