GetsDrawnDotCom

GetsDrawn DotCom

This is a python script to generate the website GetsDrawn. It takes data from /r/RedditGetsDrawn and makes something awesome.

The script has envolved and been rewritten several times.

The first script for rgdsnatch was written after I got banned from posting my artwork on /r/RedditGetsDrawn. The plan was to create a new site that displayed stuff from /r/RedditGetsDrawn.

Currently it gets the most recent 25 items on redditgetsdrawn, and saves it to a folder. The script looks at the newest 25 reference photos on RedditGetsDrawn. It focuses only on jpeg/png images and ignores and links to none .jpg or .png ending files. It is needed to instead of ignoring them files - get the image or images in some cases, from the link. The photos are always submitted from imgur. Still filter out the i.imgur files, but take the links and filter them through a python imgur module returning the .jpeg or .png files.

This is moving forward from rgdsnatch.py because I am stuck on it.

TODO

Fix the links that don't link to png/jpeg and link to webaddress. Needs to get the images that are at that web address and embed them.

Display artwork submitted under the images.

Upload artwork to user. Sends them a message on redditgetsdrawn with links.

More pandas

Saves reference images to imgs/year/month/day/reference/username-reference.png

Saves art images to imgs/year/month/day/art/username-line-bw-colour.png

Creates index.html file with: Title of site and logo: GetsDrawn Last updated date and time.

Path of image file /imgs/year/month/day/username-reference.png. (This needs changed to just their username).

Save off .meta data from reddit of each photo, saving it to reference folder. username-yrmnthday.meta - contains info such as author, title, upvotes, downvotes. Currently saving .meta files to a meta folder - along side art and reference.

Folder sorting system of files. websitename/index.html-style.css-imgs/YEAR(15)-MONTH(2)-DAY(4)/art-reference-meta Inside art folder Currently it generates USERNAME-line/bw/colour.png 50/50 white files. Maybe should be getting art replies from reddit?

Inside reference folder Reference fold is working decent. it creates USERNAME-reference.png / jpeg files.

Currently saves username-line-bw-colour.png to imgs folder. Instead get it to save to imgs/year/month/day/usernames.png. Script checks the year/month/day and if folder isnt created, it creates it. If folder is there, exit. Maybe get the reference image and save it with the line/bw/color.pngs

The script now filters the jpeg and png image and skips links to imgur pages. This needs to be fixed by getting the images from the imgur pages. It renames the image files to the redditor username followed by a -reference tag (and ending with png of course). It opens these files up with PIL and checks the sizes. It needs to resize the images that are larger than 800px to 800px. These images need to be linked in the index.html instead of the imgur altenatives.

Instead of the jpeg/png files on imgur they are downloaded to the server with this script.

Filter through as images are getting downloaded and if it has been less than certain time or if the image has been submitted before

Extending the subreddits it gets data from to cycle though a list, run script though list of subreddits.

Browse certain days - Current day by default but option to scroll through other days.

Filters - male/female/animals/couples etc Function that returns only male portraits. tags to add to photos. Filter images with tags

In [2]:
import os 
import requests
from bs4 import BeautifulSoup
import re
import json
import time
import praw
import dominate
from dominate.tags import * 
from time import gmtime, strftime
#import nose
#import unittest
import numpy as np
import pandas as pd
from pandas import *
from PIL import Image
from pprint import pprint
#import pyttsx
import shutil
import getpass
In [3]:
hosnam = getpass.getuser()
In [4]:
gtsdrndir = ('/home/' + hosnam + '/getsdrawndotcom/')
In [5]:
gtsdrndir
Out[5]:
'/home/wcmckee/getsdrawndotcom/'
In [7]:
if os.path.isdir(gtsdrndir) == True:
    print 'its true'
else:
    print 'its false'
    os.mkdir(gtsdrndir)
its false
In [9]:
os.chdir(gtsdrndir)
In [10]:
r = praw.Reddit(user_agent='getsdrawndotcom')
In [11]:
#getmin = r.get_redditor('itwillbemine')
In [12]:
#mincom = getmin.get_comments()
In [13]:
#engine = pyttsx.init()

#engine.say('The quick brown fox jumped over the lazy dog.')
#engine.runAndWait()
In [14]:
#shtweet = []
In [15]:
#for mi in mincom:
#    print mi
#    shtweet.append(mi)
In [16]:
bodycom = []
bodyicv = dict()
In [17]:
#beginz = pyttsx.init()
In [18]:
#for shtz in shtweet:
#    print shtz.downs
#    print shtz.ups
#    print shtz.body
#    print shtz.replies
    #beginz.say(shtz.author)
    #beginz.say(shtz.body)
    #beginz.runAndWait()
    
#    bodycom.append(shtz.body)
    #bodyic
In [19]:
#bodycom 
In [20]:
getnewr = r.get_subreddit('redditgetsdrawn')
In [21]:
rdnew = getnewr.get_new()
In [22]:
lisrgc = []
lisauth = []
In [23]:
for uz in rdnew:
    #print uz
    lisrgc.append(uz)
In [24]:
gtdrndic = dict()
In [30]:
imgdir = (gtsdrndir + 'imgs')
In [35]:
imgdir
Out[35]:
'/home/wcmckee/getsdrawndotcom/imgs'
In [37]:
if os.path.isdir(imgdir) == True:
    print 'its true'
else:
    print 'its false'
    os.mkdir(imgdir)
its false
In [38]:
artlist = os.listdir(imgdir)
In [39]:
from time import time
In [40]:
yearz = strftime("%y", gmtime())
monthz = strftime("%m", gmtime())
dayz = strftime("%d", gmtime())


#strftime("%y %m %d", gmtime())
In [53]:
yrzpat = (imgdir + yearz)
monzpath = (yrzpat + '/' + monthz)
dayzpath = (monzpath + '/' + dayz)
rmgzdays = (dayzpath + '/reference')
imgzdays = (dayzpath + '/art')
metzdays = (dayzpath + '/meta')

repathz = (imgdir + '/' + yearz + '/' + monthz + '/' + dayz + '/')
In [54]:
repathz
Out[54]:
'/home/wcmckee/getsdrawndotcom/imgs/15/01/31/'
In [55]:
imgzdays
Out[55]:
'/home/wcmckee/getsdrawndotcom/imgs15/01/31/art'
In [56]:
repathz
Out[56]:
'/home/wcmckee/getsdrawndotcom/imgs/15/01/31/'
In [57]:
def ospacheck():
    if os.path.isdir(imgdir + yearz) == True:
        print 'its true'
    else:
        print 'its false'
        os.mkdir(imgdir + yearz)
    
In [58]:
ospacheck()
its false
In [47]:
#if os.path.isdir(imgzdir + yearz) == True:
#    print 'its true'
#else:
#    print 'its false'
#    os.mkdir(imgzdir + yearz)
In [48]:
lizmon = ['monzpath', 'dayzpath', 'imgzdays', 'rmgzdays', 'metzdays']

Something is wrong with the script and it's no longer creating these dir in the correct folder. How did this break? Fixed that but problems with it Getting error: OSError: [Errno 17] File exists: '/home/wcmckee/getsdrawndotcom/imgs/15/01' If the file exists it should be skipping over it, thats why it has the os.path.isdir == True: print its true else print its false, and make the dir

In [231]:
if os.path.isdir(monzpath) == True:
    print 'its true'
else:
    print 'its false'
    #os.mkdir('/home/wcmckee/getsdrawndotcom/' + monzpath)
its true
In [230]:
if os.path.isdir(dayzpath) == True:
    print 'its true'
else:
    print 'its false'
    os.mkdir(dayzpath)

if os.path.isdir(imgzdays) == True:
    print 'its true'
else:
    print 'its false'
    os.mkdir(imgzdays)
    
if os.path.isdir(rmgzdays) == True:
    print 'its true'
else:
    print 'its false'
    os.mkdir(rmgzdays)
    
if os.path.isdir(metzdays) == True:
    print 'its true'
else:
    print 'its false'
    os.mkdir(metzdays)
its true
its true
its true
its true
In [158]:
#for liz in lizmon:
#    if os.path.isdir(liz) == True:
##        print 'its true'
 #   else:
#        print 'its false'
#        os.mkdir(liz)
In [159]:
fullhom = ('/home/wcmckee/getsdrawndotcom/')
In [160]:
#artlist
In [161]:
httpad = ('http://getsdrawn.com/imgs')
In [162]:
#im = Image.new("RGB", (512, 512), "white")
#im.save(file + ".thumbnail", "JPEG")
In [163]:
rmgzdays = (dayzpath + '/reference')
imgzdays = (dayzpath + '/art')
metzdays = (dayzpath + '/meta')
In [165]:
os.chdir(metzdays)
In [166]:
metadict = dict()

if i save the data to the file how am i going to get it to update as the post is archieved. Such as up and down votes.

In [167]:
for lisz in lisrgc:
    metadict.update({'up': lisz.ups})
    metadict.update({'down': lisz.downs})
    metadict.update({'title': lisz.title})
    metadict.update({'created': lisz.created})
    #metadict.update({'createdutc': lisz.created_utc})
    #print lisz.ups
    #print lisz.downs
    #print lisz.created
    #print lisz.comments
In [168]:
metadict
Out[168]:
{'created': 1420509834.0,
 'down': 0,
 'title': u"I'm getting married and would love a portrait of Myself and my Fiancee",
 'up': 0}

Need to save json object.

Dict is created but it isnt saving. Looping through lisrgc twice, should only require the one loop.

Cycle through lisr and append to dict/concert to json, and also cycle through lisr.author meta folders saving the json that was created.

In [170]:
for lisr in lisrgc:
    gtdrndic.update({'title': lisr.title})
    lisauth.append(str(lisr.author))
    for osliz in os.listdir(metzdays):
        with open(str(lisr.author) + '.meta', "w") as f:
            rstrin = lisr.title.encode('ascii', 'ignore').decode('ascii')
            #print matdict
            #metadict = dict()
            #for lisz in lisrgc:
            #    metadict.update({'up': lisz.ups})
            #    metadict.update({'down': lisz.downs})
            #    metadict.update({'title': lisz.title})
            #    metadict.update({'created': lisz.created})
            f.write(rstrin)
In [171]:
#matdict

I have it creating a meta folder and creating/writing username.meta files. It wrote 'test' in each folder, but now it writes the photo author title of post.. the username/image data. It should be writing more than author title - maybe upvotes/downvotes, subreddit, time published etc.

In [172]:
#os.listdir(dayzpath)

Instead of creating these white images, why not download the art replies of the reference photo.

In [173]:
#for lisa in lisauth:
#    #print lisa + '-line.png'
#    im = Image.new("RGB", (512, 512), "white")
#    im.save(lisa + '-line.png')
#    im = Image.new("RGB", (512, 512), "white")
#    im.save(lisa + '-bw.png')

    #print lisa + '-bw.png'
#    im = Image.new("RGB", (512, 512), "white")
#    im.save(lisa + '-colour.png')

    #print lisa + '-colour.png'
In [174]:
os.listdir('/home/wcmckee/getsdrawndotcom/imgs')
Out[174]:
['getsdrawn-bw.png', '12', '15', '14']
In [175]:
#lisauth

I want to save the list of usernames that submit images as png files in a dir. Currently when I call the list of authors it returns Redditor(user_name='theusername'). I want to return 'theusername'. Once this is resolved I can add '-line.png' '-bw.png' '-colour.png' to each folder.

In [176]:
#lisr.author
In [177]:
namlis = []
In [178]:
opsinz = open('/home/wcmckee/visignsys/index.meta', 'r')
panz = opsinz.read()
In [180]:
os.chdir(rmgzdays)

Filter the non jpeg/png links. Need to perform request or imgur api to get the jpeg/png files from the link. Hey maybe bs4?

In [180]:
 
In [181]:
from imgurpython import ImgurClient
In [182]:
opps = open('/home/wcmckee/ps.txt', 'r')
opzs = open('/home/wcmckee/ps2.txt', 'r')
oprd = opps.read()
opzrd = opzs.read()
In [183]:
client = ImgurClient(oprd, opzrd)

# Example request
#items = client.gallery()
#for item in items:
#    print(item.link)
    

#itz = client.get_album_images()
In [184]:
galim = client.get_image('SBaV275')
In [185]:
galim.size
Out[185]:
1947098
In [186]:
gelim = client.get_album_images('LTDJ9')
In [187]:
gelim
Out[187]:
[<imgurpython.imgur.models.image.Image instance at 0xafa02120>,
 <imgurpython.imgur.models.image.Image instance at 0xaf9f18c8>,
 <imgurpython.imgur.models.image.Image instance at 0xaf9f1238>,
 <imgurpython.imgur.models.image.Image instance at 0xaf9f1d78>,
 <imgurpython.imgur.models.image.Image instance at 0xaf9f1b70>,
 <imgurpython.imgur.models.image.Image instance at 0xaf9f1d28>,
 <imgurpython.imgur.models.image.Image instance at 0xaf9f1a80>,
 <imgurpython.imgur.models.image.Image instance at 0xaf9f1e40>]
In [188]:
from urlparse import urlparse
In [189]:
linklis = []

I need to get the image ids from each url. Strip the http://imgur.com/ from the string. The gallery id is the random characters after. if it's an album a is added. if multi imgs then , is used to seprate.

Doesnt currently work.

Having problems with mixed /a/etwet and wetfwet urls. Using .strip('/') to remove forward slash in front of path.

In [190]:
pathlis = []
In [191]:
for rdz in lisrgc:
    if 'http://imgur.com/' in rdz.url:
        print rdz.url
        parsed = urlparse(rdz.url)
        print parsed.path.strip('/')
        pathlis.append(parsed.path.strip('/'))
        #for pared in parsed.path:
        #    print pared.strip('/')
        #itgar = client.gallery_item(parsed.path.strip('/'))
        #itz = client.get_album_images(parsed.path.strip('a/'))
#        reimg = requests.get(rdz.url)
##        retxt = reimg.text
#        souptxt = BeautifulSoup(''.join(retxt))
#        soupurz = souptxt.findAll('img')
#        for soupuz in soupurz:
#            imgurl = soupuz['src']
#            print imgurl
#            linklis.append(imgurl)
            
            #try:
            #    imzdata = requests.get(imgurl)
http://imgur.com/Bmr5mit
Bmr5mit
http://imgur.com/putrdyx
putrdyx
http://imgur.com/xXI94VS
xXI94VS
http://imgur.com/sEJtVao
sEJtVao
http://imgur.com/a/h7aS9
a/h7aS9
http://imgur.com/m9agv3W
m9agv3W
http://imgur.com/a/SBSkJ
a/SBSkJ
http://imgur.com/OTRADPu
OTRADPu
http://imgur.com/Z1XdGv3
Z1XdGv3
http://imgur.com/pgqisMy
pgqisMy
http://imgur.com/9p61Jtb
9p61Jtb
http://imgur.com/a/IkZ37
a/IkZ37
http://imgur.com/Wh7hbU2
Wh7hbU2
http://imgur.com/Jr85F4I
Jr85F4I
http://imgur.com/VJOonmM
VJOonmM
In [192]:
pathlis
Out[192]:
[u'Bmr5mit',
 u'putrdyx',
 u'xXI94VS',
 u'sEJtVao',
 u'a/h7aS9',
 u'm9agv3W',
 u'a/SBSkJ',
 u'OTRADPu',
 u'Z1XdGv3',
 u'pgqisMy',
 u'9p61Jtb',
 u'a/IkZ37',
 u'Wh7hbU2',
 u'Jr85F4I',
 u'VJOonmM']
In [193]:
noalis = []
In [194]:
for pathl in pathlis:
    if 'a/' in pathl:
        print 'a found'
    else:
        noalis.append(pathl)
a found
a found
a found
In [195]:
#if 'a/' in pathlis:
#    print 'a found'
#else:
#    noalis.append(pathlis)
In [196]:
for noaz in noalis:
    print noaz
    #itgar = client.gallery_item()
Bmr5mit
putrdyx
xXI94VS
sEJtVao
m9agv3W
OTRADPu
Z1XdGv3
pgqisMy
9p61Jtb
Wh7hbU2
Jr85F4I
VJOonmM
In [197]:
linklis
Out[197]:
[]
In [198]:
if '.jpg' in linklis:
    print 'yes'
else:
    print 'no'
no
In [199]:
#panz()
for rdz in lisrgc:
    (rdz.title)
    #a(rdz.url)
    if 'http://i.imgur.com' in rdz.url:
        #print rdz.url
        print (rdz.url)
        url = rdz.url
        response = requests.get(url, stream=True)
        with open(str(rdz.author) + '-reference.png', 'wb') as out_file:
            shutil.copyfileobj(response.raw, out_file)
            del response
http://i.imgur.com/zUmgwd7.jpg
http://i.imgur.com/iLDvh9P.jpg?1
http://i.imgur.com/kI7enKC.jpg
---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-199-da5355938f18> in <module>()
      9         response = requests.get(url, stream=True)
     10         with open(str(rdz.author) + '-reference.png', 'wb') as out_file:
---> 11             shutil.copyfileobj(response.raw, out_file)
     12             del response

/usr/lib/python2.7/shutil.pyc in copyfileobj(fsrc, fdst, length)
     47     """copy data from file-like object fsrc to file-like object fdst"""
     48     while 1:
---> 49         buf = fsrc.read(length)
     50         if not buf:
     51             break

/usr/lib/python2.7/dist-packages/urllib3/response.pyc in read(self, amt, decode_content, cache_content)
    172             else:
    173                 cache_content = False
--> 174                 data = self._fp.read(amt)
    175                 if amt != 0 and not data:  # Platform-specific: Buggy versions of Python.
    176                     # Close the connection when no data is returned

/usr/lib/python2.7/httplib.pyc in read(self, amt)
    565         # connection, and the user is reading more bytes than will be provided
    566         # (for example, reading in 1k chunks)
--> 567         s = self.fp.read(amt)
    568         if not s and amt:
    569             # Ideally, we would raise IncompleteRead if the content-length

/usr/lib/python2.7/socket.pyc in read(self, size)
    378                 # fragmentation issues on many platforms.
    379                 try:
--> 380                     data = self._sock.recv(left)
    381                 except error, e:
    382                     if e.args[0] == EINTR:

KeyboardInterrupt: 
In [200]:
apsize = []
In [201]:
aptype = []
In [202]:
basewidth = 600
In [203]:
imgdict = dict()
In [205]:
for rmglis in os.listdir(rmgzdays):
    #print rmglis
    im = Image.open(rmglis)
    #print im.size
    imgdict.update({rmglis : im.size})
    #im.thumbnail(size, Image.ANTIALIAS)
    #im.save(file + ".thumbnail", "JPEG")
    apsize.append(im.size)
    aptype.append(rmglis)
In [206]:
#for imdva in imgdict.values():
    #print imdva
    #for deva in imdva:
        #print deva
     #   if deva < 1000:
      #      print 'omg less than 1000'
       # else:
        #    print 'omg more than 1000'
         #   print deva / 2
            #print imgdict.values
            # Needs to update imgdict.values with this new number. Must halve height also.
In [207]:
#basewidth = 300
#img = Image.open('somepic.jpg')
#wpercent = (basewidth/float(img.size[0]))
#hsize = int((float(img.size[1])*float(wpercent)))
#img = img.resize((basewidth,hsize), PIL.Image.ANTIALIAS)
#img.save('sompic.jpg')
In [208]:
#os.chdir(metzdays)
In [208]:
 
In [209]:
#for numz in apsize:
#    print numz[0]
 #   if numz[0] > 800:
#        print ('greater than 800')
#    else:
#        print ('less than 800!')
In [210]:
reliz = []
In [212]:
for refls in os.listdir(rmgzdays):
    #print rmgzdays + refls
    reliz.append(rmgzdays + '/' + refls)
In [213]:
reliz
Out[213]:
['/home/wcmckee/getsdrawndotcom/imgs/15/01/06/reference/equinox5005-reference.png',
 '/home/wcmckee/getsdrawndotcom/imgs/15/01/06/reference/lennon3862-reference.png',
 '/home/wcmckee/getsdrawndotcom/imgs/15/01/06/reference/Bitches_BeCrazy-reference.png']
In [214]:
aptype
Out[214]:
['equinox5005-reference.png',
 'lennon3862-reference.png',
 'Bitches_BeCrazy-reference.png']
In [215]:
opad = open('/home/wcmckee/ad.html', 'r')
In [216]:
opred = opad.read()
In [217]:
str2 = opred.replace("\n", "")
In [218]:
str2
Out[218]:
'<script async src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script><!-- header --><ins class="adsbygoogle"     style="display:inline-block;width:970px;height:250px"     data-ad-client="ca-pub-2716205862465403"     data-ad-slot="3994067148"></ins><script>(adsbygoogle = window.adsbygoogle || []).push({});</script>'
In [219]:
doc = dominate.document(title='GetsDrawn')

with doc.head:
    link(rel='stylesheet', href='style.css')
    script(type ='text/javascript', src='script.js')
    str(str2)
    
    with div():
        attr(cls='header')
        h1('GetsDrawn')
        p(img('imgs/getsdrawn-bw.png', src='imgs/getsdrawn-bw.png'))
        #p(img('imgs/15/01/02/ReptileLover82-reference.png', src= 'imgs/15/01/02/ReptileLover82-reference.png'))
        h1('Updated ', strftime("%a, %d %b %Y %H:%M:%S +0000", gmtime()))
        p(panz)
        p(bodycom)
    
    

with doc:
    with div(id='body').add(ol()):
        for rdz in reliz:
            #h1(rdz.title)
            #a(rdz.url)
            #p(img(rdz, src='%s' % rdz))
            #print rdz
            p(img(rdz, src = rdz))
            p(rdz)


                
            #print rdz.url
            #if '.jpg' in rdz.url:
            #    img(rdz.urlz)
            #else:
            #    a(rdz.urlz)
            #h1(str(rdz.author))
            
            #li(img(i.lower(), src='%s' % i))

    with div():
        attr(cls='body')
        p('GetsDrawn is open source')
        a('https://github.com/getsdrawn/getsdrawndotcom')
        a('https://reddit.com/r/redditgetsdrawn')

#print doc
In [220]:
docre = doc.render()
In [221]:
#s = docre.decode('ascii', 'ignore')
In [222]:
yourstring = docre.encode('ascii', 'ignore').decode('ascii')
In [223]:
indfil = ('/home/wcmckee/getsdrawndotcom/index.html')
In [224]:
mkind = open(indfil, 'w')
mkind.write(yourstring)
mkind.close()
In [225]:
#os.system('scp -r /home/wcmckee/getsdrawndotcom/ wcmckee@getsdrawn.com:/home/wcmckee/getsdrawndotcom')
In [226]:
#rsync -azP source destination
In [80]:
#updatehtm = raw_input('Update index? Y/n')
#updateref = raw_input('Update reference? Y/n')

#if 'y' or '' in updatehtm:
#    os.system('scp -r /home/wcmckee/getsdrawndotcom/index.html wcmckee@getsdrawn.com:/home/wcmckee/getsdrawndotcom/index.html')
#elif 'n' in updatehtm:
#    print 'not uploading'
#if 'y' or '' in updateref:
#    os.system('rsync -azP /home/wcmckee/getsdrawndotcom/ wcmckee@getsdrawn.com:/home/wcmckee/getsdrawndotcom/')
In [81]:
os.system('scp -r /home/wcmckee/getsdrawndotcom/index.html wcmckee@getsdrawn.com:/home/wcmckee/getsdrawndotcom/index.html')
Out[81]:
0
In [553]:
#os.system('scp -r /home/wcmckee/getsdrawndotcom/style.css wcmckee@getsdrawn.com:/home/wcmckee/getsdrawndotcom/style.css')
In [553]:
 
In [321]:
 
In [138]:
 
In [138]:
 
In [ ]:
 

Comments

Comments powered by Disqus