gmail-parser.py failure

[hakerdefo]

@Snap is right! Screw Python for now and switch to Ruby 

First make sure following packages are installed,

Code: [Select]

ruby
libruby

Gmail by default blocks all non-Google applications from accessing your account. There are two things you can do here to make this script work with Gmail.
First Method (recommended) :-
Generate app password for this script and use that generated password in the script. More information here,

Sign in using App Passwords
Second Method :-
Change your Goolge account setting to allow non-Google apps access to your account. More information here,
Allowing less secure apps to access your account

Next copy the following script with the name 'check-gmail.rb' somewhere in your $PATH,

Code: [Select]

#!/usr/bin/ruby
require 'net/imap'
IMAP_USER = "your_gmail_username"
IMAP_PASS = "your_gmail_password"
IMAP_SERVER = "imap.gmail.com"
SSL = true
PORT = 993
IS_EXCHANGE = false
IMAP_FOLDER = "INBOX"
CONKYFIED = true
NEW_COLOR = "green"
SENDER_COLOR = "grey"
SUBJ_COLOR = "white"

def conkify(text,color)
    text = "${color #{color}}#{text} $color"
    return text
end

def get_messages(conn)
    imap = conn
    imap.search(["NOT", "DELETED", "UNSEEN"]).each do |message_id|
        envelope = imap.fetch(message_id, "ENVELOPE")[0].attr["ENVELOPE"]
        from, subject = envelope.from[0].name == nil ? envelope.from[0].mailbox : envelope.from[0].name, envelope.subject
        CONKYFIED == true ? puts("#{conkify(from,SENDER_COLOR)} $alignr#{conkify(subject,SUBJ_COLOR)}") : puts("#{from}  #{subject}")
    end
end

imap = Net::IMAP.new(IMAP_SERVER,port = PORT, usessl = SSL)
imap.login(IMAP_USER,IMAP_PASS)
imap.examine(IMAP_FOLDER)
IS_EXCHANGE == true ? counts = imap.search(["NOT", "DELETED", "UNSEEN"]).size : counts = imap.status(IMAP_FOLDER, ["UNSEEN"])["UNSEEN"]
counts > 0 ? ( puts("New Messages: #{CONKYFIED == true ? conkify(counts,NEW_COLOR) : counts}"); get_messages(imap) ) : ( puts("No new messages") )
imap.disconnect
exit


You'll need to modify the following in the script before using it,

Code: [Select]

IMAP_USER = "your_gmail_username"
IMAP_PASS = "your_gmail_password"

To modify the color output of the script shown in conky, modify the following fields to your liking in the script,

Code: [Select]

NEW_COLOR = "green"
SENDER_COLOR = "grey"
SUBJ_COLOR = "white"

If you wish to use the script in terminal instead of conky, change the following in the script from,

Code: [Select]

CONKYFIED = true

to

Code: [Select]

CONKYFIED = false

Make sure to give the script correct permissions,

Code: [Select]

chmod 755 /path/to/check-gmail.rb

And call it like following from conky,

Code: [Select]

execpi 600 /path/to/check-gmail.rb

This is tested okay & satisfaction guaranteed!!!

Cheers!!!

[PackRat]

@PackRat @VastOne
You've got following installed?

Code: [Select]

python3-openssl
python3-requests
python3-urllib3

Install if missing and re-run the script!

Cheers!!!

No change

[hakerdefo]

Time to put this thread to rest! Why? Cause my friend Snap could be right when he said,

Folks, AFAIK there's currently something libssl breaking python. Might be that driving you mad?

What's the proof I hear you say?

Well, I fired-up Semplice workstation (based on Debian stable) in live session. Took the 'check-gmail.py' posted earlier in this thread by VastOne and installed following packages,

Code: [Select]

python-urllib3
python-requests
python-feedparser

And run the 'check-gmail.py' script thus,

Code: [Select]

python check-gmail.py "gmail_username" "gmail_password" "3"

The result? Here you go,

The username & password are hidden in the image for obvious reasons but you know what? It Fu***ng works 

Now why oh why this doesn't work in Sid? As Snap mentioned there could be some bug out there somewhere in the Python land that's why!
What to do in the mean time? Use the ruby script I posted previously. It's small, doesn't have any dependencies outside standard Ruby install and it's conky friendly! What else do you want 

Cheers!!!

[VastOne]

While I am stoked beyond belief for your help hakerdefo and love the ruby solution... it is a far cry from what I have been used to and perhaps you might want to improve it

Here is what I see:

After working all day I have 36 new messages on Gmail and this is how the new Ruby shows it


 


This is how the old shows it



What would be need should be explained in the old script...

But, if no one can solve this the third and likable option is this bash option that simply shows how how may messages I have on gmail

mail-notify (make it executable)

Code: [Select]

#!/bin/bash

gmail_login="your_gmail_login_name" #login name
gmail_password="your_gmail_login_password" #pass

dane="$(wget --secure-protocol=TLSv1 --timeout=3 -t 1 -q -O - \
https://${gmail_login}:${gmail_password}@mail.google.com/mail/feed/atom \
--no-check-certificate | grep 'fullcount' \
| sed -e 's/.*<fullcount>//;s/<\/fullcount>.*//' 2>/dev/null)"

if [ -z "$dane" ]; then
echo "No connection"
else
echo "$dane message(s)"
fi
run it in conky like this:

Code: [Select]

${color #E2CF3B} ${execi 20 /home/vastone/mail-notify}
It looks like this:


 

[hakerdefo]

Okay, one more go at the python.

Gmail by default blocks all non-Google applications from accessing your account. There are two methods to make this script work with Gmail.

First Method (recommended) :-

Generate app password for this script and use that generated password in the script. More information here,
Sign in using App Passwords

Second Method :-

Change your Goolge account setting to allow non-Google apps access to your account. More information here,
Allowing less secure apps to access your account

Open this script in your text-editor,

Code: [Select]

#!/usr/bin/env python2
import sys
import socket
import urllib
import errno
import os
import signal
from xml.dom import minidom
from functools import wraps

username = "your_gmail_username"
password = "your_gmail_password"
label    = str(sys.argv[1])
limit    = int(sys.argv[2])

class TimeoutError(Exception):
    pass

def timeout(seconds=10, error_message=os.strerror(errno.ETIME)):
    def decorator(func):
        def _handle_timeout(signum, frame):
            raise TimeoutError(error_message)

        @wraps(func)
        def wrapper(*args, **kwargs):
            signal.signal(signal.SIGALRM, _handle_timeout)
            signal.alarm(seconds)
            try:
                result = func(*args, **kwargs)
            finally:
                signal.alarm(0)
            return result

        return wrapper

    return decorator

GMAIL_URL = 'https://'+urllib.quote(username)+':'+urllib.quote(password)+'@mail.google.com/mail/feed/atom/'+urllib.quote(label)+'/'

def utfEncode(string):
    return unicode(string).encode('utf-8')

def printEmails(emails):
    for email in emails['emails']:
        print 'Sender: %s' %(utfEncode(email['address']))
        print 'Subject: %s' %utfEncode(email['title'])
        print 'Summary: %s\n' %utfEncode(email['summary'])

@timeout(10)
def getGmail(label,maxCount):
    url = GMAIL_URL
    try:
        xml = urllib.urlopen(url)
        dom = minidom.parse(xml)
    except:
        print '\nError: Could not get emails\n'
        exit()
    mail = []
    for node in dom.getElementsByTagName('entry'):
        if maxCount == 0:
          break
        author = node.getElementsByTagName('author')[0]
        mail.append({
            'title'  : node.getElementsByTagName('title')[0].firstChild.data,
            'summary': node.getElementsByTagName('summary')[0].firstChild.data,
            'sender' : author.getElementsByTagName('name')[0].firstChild.data,
            'address': author.getElementsByTagName('email')[0].firstChild.data
        })
        maxCount -= 1
    count = dom.getElementsByTagName('fullcount')[0].firstChild.data
    return {
        'emails'    : mail,
        'labelname' : label,
        'labelcount': count
    }

printEmails(getGmail(label,limit))


And change the following fields in the script appropriately,

Code: [Select]

username = "your_gmail_username"
password = "your_gmail_password"

Save the script somewhere in your $PATH with the name 'check-gmail.py' and give it execute permission,

Code: [Select]

chmod 755 /path/to/check-gmail.py

Open '.conkyrc' in your text-editor and add entry for 'check-gmail.py' like this,

Code: [Select]

execi 300 python2 /path/to/check-gmail.py INBOX 5

IMPORTANT:- Don't change 'python2' and 'INBOX' from the above snippet. The script won't work if you change either of them. Of-course you can change update interval from 300 seconds to some value of your choice. '5' in the above snippet determines the number of unread mail the script will display. You can change it from '5' to the value of your choice.

A scrot,



This worked for me as you can see from the scrot but your mileage may vary 

This script won't work with python3. If you want to run it in the terminal, run it like this via python2,

Code: [Select]

python2 /path/to/check-gmail.py INBOX 5

Try it! What's the harm!

Cheers!!!

[VastOne]

Code: [Select]

vastone@vsido:~$ python2 ./check-gmail.py INBOX 5

Error: Could not get emails
I cannot debug anything further... this is an epic script and I hope to get it working

Well done man...

V-Ger

[hakerdefo]

Code: [Select]

vastone@vsido:~$ python2 ./check-gmail.py INBOX 5

Error: Could not get emails
I cannot debug anything further... this is an epic script and I hope to get it working

Well done man...

V-Ger

That happened to me once or twice as well. Re-run the script and it returns the results. Retry it, chances are it'll work.
P.S. I again recommend you create and use a Google apps password with this script. Not only this script but every script-app that interacts with your account. You can create as many apps password as you wish.

Sign in using App Passwords

Cheers!!!

[VastOne]

Still does not work for me.. restarted it several dozen times and even rebooted

Is there something new in the requirements from the import section?

Code: [Select]

import sys
import socket
import urllib
import errno
import os
import signal
from xml.dom import minidom
from functools import wraps

[hakerdefo]

Still does not work for me.. restarted it several dozen times and even rebooted

Is there something new in the requirements from the import section?

Code: [Select]

import sys
import socket
import urllib
import errno
import os
import signal
from xml.dom import minidom
from functools import wraps

No, it's not a missing module problem. Had it been a module missing thingy python would have mentioned it in the error. Most likely Google is blocking the log-in attempt. If you are using your Google account password in the script then you'll have to change the account setting to allow non-Google apps access to your account. More information here,
Allowing less secure apps to access your account

Cheers!!!

[hakerdefo]

The original script (gmail-parser.py) that started the thread works fine on Debian stable. So we can assume that the one or more python modules packaged in Debian unstable is/are causing these scripts to fail on Debian unstable. One idea to try and resolve this problem is to use python modules from PyPI instead of Debian unstable.
First remove all python modules you have installed to meet the script dependencies. Like python-urllib3, python-requests, python-feedparser, python-openssl, python3-openssl, python3-requests, python3-urllib3, python3-feedparser etc. from the system. Next download the following package,

https://bootstrap.pypa.io/get-pip.py

cd to the download directory and run following commands in the exact order,

Code: [Select]

sudo python2 get-pip.py
sudo python3 get-pip.py

Then use pip to install modules required by a python script. For example if a python2 script requires 'feedparser' module, you can install it like this,

Code: [Select]

sudo pip2 install feedparser
And if a python3 script requires module 'feedparser', you can do,

Code: [Select]

sudo pip3 install feedparser
For more in-depth look at pip read this.

http://matthew-brett.github.io/pydagogue/installing_on_debian.html

Cheers!!!

[VastOne]

Hi hakerdefo

It is definitely urllib/libssl that snap pointed out that is the issue. I am now on the machine that has sid running and just did an update and was told that libssl would be updated and that a bug is going to break python... nice.

I will try your new solutions but I already have my google acct setup as Access for less secure apps    and have verified with earlier scripts (ruby one) that it works as advertised.. It must be noted that I cannot change my google acct to any other setting. IOW, I do not have the option to go with different passwords for a more secure environment

I will get back with feedback

[VastOne]

More weirdness...

I have disabled the check-gmail.py line from my conky script but yet the script just keeps on running... I do not see it in Task Manager or in top as a process running

The message

Error:Could not get emails

is explicit to the check-gmail.py script and when I delete the contents of that file and save it, only then does conky remove that message from my conky widget


 

[hakerdefo]

Hi hakerdefo

It is definitely urllib/libssl that snap pointed out that is the issue. I am now on the machine that has sid running and just did an update and was told that libssl would be updated and that a bug is going to break python... nice.

I will try your new solutions but I already have my google acct setup as Access for less secure apps and have verified with earlier scripts (ruby one) that it works as advertised.. It must be noted that I cannot change my google acct to any other setting. IOW, I do not have the option to go with different passwords for a more secure environment

I will get back with feedback

Hi there,
I would suggest you should first turn-off access to less secure apps here,

http://www.google.com/settings/security/lesssecureapps

Then whenever you need to provide your Google password to any app or script go the following page and create app password for that app/script,

https://security.google.com/settings/security/apppasswords

NOTE: Google will display the 16 character long app password only once when you create it.

You can always go to 'App passwords' page and remove an 'app password' whenever you want.
You can generate as many apps passwords as you wish so create-use a different app password for every application/script you use.

NOTE: You won't be able to create an App password if you have turned-on 'Access for less secure apps'.

Cheers!!!

[hakerdefo]

More weirdness...

I have disabled the check-gmail.py line from my conky script but yet the script just keeps on running... I do not see it in Task Manager or in top as a process running

The message

Error:Could not get emails

is explicit to the check-gmail.py script and when I delete the contents of that file and save it, only then does conky remove that message from my conky widget

Strange indeed! As I've mentioned in my second last post you can use python modules from PyPI instead of Debian unstable whenever a script needs a module.

http://vsido.org/index.php?topic=1222.msg13817#msg13817

Cheers!!!

[VastOne]

I cannot access to setup app passwords even after having less secure apps turned off for 24 hours

Google sucks with a lot of their bullshit and I am tired of it all

I appreciate all your help and would like to see this script (check-gmail.py) working but my patience has run out

Hopefully in the long run the original issue with python and urllib/libssl will work itself out and the original gmail-parser.py can be used again