Mailing List Archive

What's the best way to write this regular expression?
I sort of have to work with what the website gives me (as you'll see below), but today I encountered an exception to my RE. Let me just give all the specific information first. The point of my script is to go to the specified URL and extract song information from it.

This is my RE:

song_pattern = re.compile(r'([0-9]{1,2}:[0-9]{2} [a|p].m.).*?<a.*?>(.*?)</a>.*?<a.*?>(.*?)</a>', re.DOTALL)

This is how the website is formatted:

4:25 p.m.
</div><div class="cmPlaylistContent"><strong><a href="/lsp/t24435/">AP TX SOC CPAS TRF</a></strong><br /><br /></div></li><li ><div class="cmPlaylistTime">

4:21 p.m.
</div><div class="cmPlaylistContent"><strong><a href="/lsp/t7672/">No One Else On Earth</a></strong><br /><a href="/lsp/a1924/">Wynonna</a><br /></div></li><li ><div class="cmPlaylistTime">

4:19 p.m.
</div><div class="cmPlaylistImage"><img src="http://media.cmgdigital.com/shared/amg/pic200/drp100/p109/p10901ruw7x_r85x85.jpg?998f84231a014ed68123ddb508af9480570dc122" alt="Moe Bandy" class="cmDarkBoxShadow cmPhotoBorderWhite"/></div><div class="cmPlaylistContent"><strong><a href="/lsp/t15101/">It&#39; A Cheating Situation</a></strong><br /><a href="/lsp/a5307/">Moe Bandy</a><br /><span class="sprite iconVoteUp">Votes&nbsp;&nbsp;(1) </span></div></li><li ><div class="cmPlaylistTime">

4:15 p.m.
</div><div class="cmPlaylistImage"><img src="http://media.cmgdigital.com/shared/amg/pic200/drp700/p744/p74493d85qy_r85x85.jpg?998f84231a014ed68123ddb508af9480570dc122" alt="Reba McEntire" class="cmDarkBoxShadow cmPhotoBorderWhite"/></div><div class="cmPlaylistContent"><strong><a href="/lsp/t14437/">Somebody Should Leave</a></strong><br /><a href="/lsp/a396/">REBA McENTIRE</a> & <a href="/lsp/a5765/">LINDA DAVIS</a><br /></div></li><li ><div class="cmPlaylistTime">

There's something of a pattern, although it's not always perfect. The time is listed first, and then the song information in <a> tags. However, in this particular case, you can see that for the 4:25pm entry, "AP TX SOC CPAS TRF" is extracted for the song title, and then the RE skips to the next entry in order to find the next <a> tags, which is actually the name of the next song in the list, instead of being the artist as normal. (Of course, I have no idea what AP TX SOC CPAS TRF is anyway. Usually the website doesn't list commercials or anomalies like that.)

So my first question is basic: am I even extracting the information properly? It works almost all the time, but because the website is such a mess, I pretty much have to rely on the tags being in the proper places (as they were NOT in this case!).

The second question is, to fix the above problem, would it be sufficient to rewrite my RE so that it has to find all of the specified information, i.e. a time followed by two <a> entries, BEFORE it moves on to finding the next time? I think that would have caused it to skip the 4:25 entry above, and only extract entries that have a time followed by two <a> entries (song and artist).

If this is possible, how do I rewrite it so that it has to match all the conditions without skipping over the next time entry in order to do so?

Thanks.
--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
On Tue, Mar 6, 2012 at 2:43 PM, John Salerno <johnjsal@gmail.com> wrote:
> I sort of have to work with what the website gives me (as you'll see below), but today I encountered an exception to my RE. Let me just give all the specific information first. The point of my script is to go to the specified URL and extract song information from it.
>
> This is my RE:
>
> song_pattern = re.compile(r'([0-9]{1,2}:[0-9]{2} [a|p].m.).*?<a.*?>(.*?)</a>.*?<a.*?>(.*?)</a>', re.DOTALL)

I would advise against using regular expressions to "parse" HTML:
http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags

lxml is a popular choice for parsing HTML in Python: http://lxml.de

Cheers,
Chris
--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
On Tuesday, March 6, 2012 4:52:10 PM UTC-6, Chris Rebert wrote:
> On Tue, Mar 6, 2012 at 2:43 PM, John Salerno <johnjsal@gmail.com> wrote:
> > I sort of have to work with what the website gives me (as you'll see below), but today I encountered an exception to my RE. Let me just give all the specific information first. The point of my script is to go to the specified URL and extract song information from it.
> >
> > This is my RE:
> >
> > song_pattern = re.compile(r'([0-9]{1,2}:[0-9]{2} [a|p].m.).*?<a.*?>(.*?)</a>.*?<a.*?>(.*?)</a>', re.DOTALL)
>
> I would advise against using regular expressions to "parse" HTML:
> http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags
>
> lxml is a popular choice for parsing HTML in Python: http://lxml.de
>
> Cheers,
> Chris

Thanks, that was an interesting read :)

Anything that allows me NOT to use REs is welcome news, so I look forward to learning about something new! :)
--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
On Tuesday, March 6, 2012 4:52:10 PM UTC-6, Chris Rebert wrote:
> On Tue, Mar 6, 2012 at 2:43 PM, John Salerno <johnjsal@gmail.com> wrote:
> > I sort of have to work with what the website gives me (as you'll see below), but today I encountered an exception to my RE. Let me just give all the specific information first. The point of my script is to go to the specified URL and extract song information from it.
> >
> > This is my RE:
> >
> > song_pattern = re.compile(r'([0-9]{1,2}:[0-9]{2} [a|p].m.).*?<a.*?>(.*?)</a>.*?<a.*?>(.*?)</a>', re.DOTALL)
>
> I would advise against using regular expressions to "parse" HTML:
> http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags
>
> lxml is a popular choice for parsing HTML in Python: http://lxml.de
>
> Cheers,
> Chris

Thanks, that was an interesting read :)

Anything that allows me NOT to use REs is welcome news, so I look forward to learning about something new! :)
--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
> Anything that allows me NOT to use REs is welcome news, so I look forward to learning about something new! :)

I should ask though...are there alternatives already bundled with Python that I could use? Now that you mention it, I remember something called HTMLParser (or something like that) and I have no idea why I never looked into that before I messed with REs.

Thanks.
--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
> Anything that allows me NOT to use REs is welcome news, so I look forward to learning about something new! :)

I should ask though...are there alternatives already bundled with Python that I could use? Now that you mention it, I remember something called HTMLParser (or something like that) and I have no idea why I never looked into that before I messed with REs.

Thanks.
--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
On Tuesday, March 6, 2012 5:05:39 PM UTC-6, John Salerno wrote:
> > Anything that allows me NOT to use REs is welcome news, so I look forward to learning about something new! :)
>
> I should ask though...are there alternatives already bundled with Python that I could use? Now that you mention it, I remember something called HTMLParser (or something like that) and I have no idea why I never looked into that before I messed with REs.
>
> Thanks.

Also, I just noticed Beautiful Soup, which seems appropriate. I suppose any will do, but knowing the pros and cons would help with a decision.
--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
On Tuesday, March 6, 2012 5:05:39 PM UTC-6, John Salerno wrote:
> > Anything that allows me NOT to use REs is welcome news, so I look forward to learning about something new! :)
>
> I should ask though...are there alternatives already bundled with Python that I could use? Now that you mention it, I remember something called HTMLParser (or something like that) and I have no idea why I never looked into that before I messed with REs.
>
> Thanks.

::sigh:: I'm having some trouble with the new Google Groups interface. It seems to double post, and in this case didn't post at all. If it did already, I apologize. I'll try to figure out what's happening, or just switch to a real newsgroup program.

Anyway, my question was about Beautiful Soup. I read on the doc page that BS uses a parser, which html.parser and lxml are. So I'm guessing the difference between them is that the parser is a little more "low level," whereas BS offers a higher level approach to using them? Is BS easier to write code with, while still using the power of lxml?
--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
On Tuesday, March 6, 2012 5:05:39 PM UTC-6, John Salerno wrote:
> > Anything that allows me NOT to use REs is welcome news, so I look forward to learning about something new! :)
>
> I should ask though...are there alternatives already bundled with Python that I could use? Now that you mention it, I remember something called HTMLParser (or something like that) and I have no idea why I never looked into that before I messed with REs.
>
> Thanks.

::sigh:: I'm having some trouble with the new Google Groups interface. It seems to double post, and in this case didn't post at all. If it did already, I apologize. I'll try to figure out what's happening, or just switch to a real newsgroup program.

Anyway, my question was about Beautiful Soup. I read on the doc page that BS uses a parser, which html.parser and lxml are. So I'm guessing the difference between them is that the parser is a little more "low level," whereas BS offers a higher level approach to using them? Is BS easier to write code with, while still using the power of lxml?
--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
On Tue, Mar 6, 2012 at 4:05 PM, John Salerno <johnjsal@gmail.com> wrote:
>> Anything that allows me NOT to use REs is welcome news, so I look forward to learning about something new! :)
>
> I should ask though...are there alternatives already bundled with Python that I could use? Now that you mention it, I remember something called HTMLParser (or something like that) and I have no idea why I never looked into that before I messed with REs.

HTMLParser is pretty basic, although it may be sufficient for your
needs. It just converts an html document into a stream of start tags,
end tags, and text, with no guarantee that the tags will actually
correspond in any meaningful way. lxml can be used to output an
actual hierarchical structure that may be easier to manipulate and
extract data from.

Cheers,
Ian
--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
Thanks. I'm thinking the choice might be between lxml and Beautiful
Soup, but since BS uses lxml as a parser, I'm trying to figure out the
difference between them. I don't necessarily need the simplest
(html.parser), but I want to choose one that is simple enough yet
powerful enough that I won't have to learn another method later.




On Tue, Mar 6, 2012 at 5:35 PM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
> On Tue, Mar 6, 2012 at 4:05 PM, John Salerno <johnjsal@gmail.com> wrote:
>>> Anything that allows me NOT to use REs is welcome news, so I look forward to learning about something new! :)
>>
>> I should ask though...are there alternatives already bundled with Python that I could use? Now that you mention it, I remember something called HTMLParser (or something like that) and I have no idea why I never looked into that before I messed with REs.
>
> HTMLParser is pretty basic, although it may be sufficient for your
> needs.  It just converts an html document into a stream of start tags,
> end tags, and text, with no guarantee that the tags will actually
> correspond in any meaningful way.  lxml can be used to output an
> actual hierarchical structure that may be easier to manipulate and
> extract data from.
>
> Cheers,
> Ian
--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
On Tue, 06 Mar 2012 15:05:39 -0800, John Salerno wrote:

>> Anything that allows me NOT to use REs is welcome news, so I look
>> forward to learning about something new! :)
>
> I should ask though...are there alternatives already bundled with Python
> that I could use? Now that you mention it, I remember something called
> HTMLParser (or something like that) and I have no idea why I never
> looked into that before I messed with REs.

import htmllib
help(htmllib)

The help is pretty minimal and technical, you might like to google on a
tutorial or two:

https://duckduckgo.com/html/?q=python%20htmllib%20tutorial

Also, you're still double-posting.


--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
> Also, you're still double-posting.

Grr. I just reported it to Google, but I think if I start to frequent the newsgroup again I'll have to switch to Thunderbird, or perhaps I'll just try switching back to the old Google Groups interface. I think the issue is the new interface.

Sorry.
--
http://mail.python.org/mailman/listinfo/python-list
RE: What's the best way to write this regular expression? [ In reply to ]
>
> > Also, you're still double-posting.
>
> Grr. I just reported it to Google, but I think if I start to frequent the
> newsgroup again I'll have to switch to Thunderbird, or perhaps I'll just
> try switching back to the old Google Groups interface. I think the issue is
> the new interface.
>
> Sorry.

Oddly, I see no double posting for this thread on my end (email list).

Ramit


Ramit Prasad | JPMorgan Chase Investment Bank | Currencies Technology
712 Main Street | Houston, TX 77002
work phone: 713 - 216 - 5423

--

This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.
--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
On 3/6/2012 6:05 PM, John Salerno wrote:
>> Anything that allows me NOT to use REs is welcome news, so I look
>> forward to learning about something new! :)
>
> I should ask though...are there alternatives already bundled with
> Python that I could use?

lxml is +- upward compatible with xml.etree in the stdlib.

--
Terry Jan Reedy

--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
On 3/6/2012 6:57 PM, John Salerno wrote:
>> Also, you're still double-posting.
>
> Grr. I just reported it to Google, but I think if I start to frequent
> the newsgroup again I'll have to switch to Thunderbird, or perhaps
> I'll just try switching back to the old Google Groups interface. I
> think the issue is the new interface.

I am not seeing the double posting, but I use Thunderbird + the
news.gmane.org mirrors of python-list and others.

--
Terry Jan Reedy

--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
In article
<12783654.1174.1331073814011.JavaMail.geo-discussion-forums@yner4>,
John Salerno <johnjsal@gmail.com> wrote:

> I sort of have to work with what the website gives me (as you'll see below),
> but today I encountered an exception to my RE. Let me just give all the
> specific information first. The point of my script is to go to the specified
> URL and extract song information from it.

Rule #1: Don't try to parse XML, HTML, or any other kind of ML with
regular expressions.

Rule #2: Use a dedicated ML parser. I like lxml (http://lxml.de/).
There's other possibilities.

Rule #3: If in doubt, see rule #1.
--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
After a bit of reading, I've decided to use Beautiful Soup 4, with
lxml as the parser. I considered simply using lxml to do all the work,
but I just got lost in the documentation and tutorials. I couldn't
find a clear explanation of how to parse an HTML file and then
navigate its structure.

The Beautiful Soup 4 documentation was very clear, and BS4 itself is
so simple and Pythonic. And best of all, since version 4 no longer
does the parsing itself, you can choose your own parser, and it works
with lxml, so I'll still be using lxml, but with a nice, clean overlay
for navigating the tree structure.

Thanks for the advice!
--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
John Salerno <johnjsal@gmail.com> writes:
> The Beautiful Soup 4 documentation was very clear, and BS4 itself is
> so simple and Pythonic. And best of all, since version 4 no longer
> does the parsing itself, you can choose your own parser, and it works
> with lxml, so I'll still be using lxml, but with a nice, clean overlay
> for navigating the tree structure.

I haven't used BS4 but have made good use of earlier versions.

Main thing to understand is that an awful lot of HTML in the real world
is malformed and will break an XML parser or anything that expects
syntactically invalid HTML. People tend to write HTML that works well
enough to render decently in browsers, whose parsers therefore have to
be tolerant of bad errors. Beautiful Soup also tries to make sense of
crappy, malformed, HTML. Partly as a result, it's dog slow compared to
any serious XML parser. But it works very well if you don't mind the
low speed.
--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
Ok, first major roadblock. I have no idea how to install Beautiful
Soup or lxml on Windows! All I can find are .tar files. Based on what
I've read, I can use the easy_setup module to install these types of
files, but when I went to download the setuptools package, it only
seemed to support Python 2.7. I'm using 3.2. Is 2.7 just the minimum
version it requires? It didn't say something like "2.7+", so I wasn't
sure, and I don't want to start installing a bunch of stuff that will
clog up my directories and not even work.

What's the best way for me to install these two packages? I've also
seen a reference to using setup.py...is that a separate package too,
or is that something that comes with Python by default?

Thanks.
--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
On Wed, Mar 7, 2012 at 1:39 PM, John Salerno <johnjsal@gmail.com> wrote:
> Ok, first major roadblock. I have no idea how to install Beautiful
> Soup or lxml on Windows! All I can find are .tar files. Based on what
> I've read, I can use the easy_setup module to install these types of
> files, but when I went to download the setuptools package, it only
> seemed to support Python 2.7. I'm using 3.2. Is 2.7 just the minimum
> version it requires? It didn't say something like "2.7+", so I wasn't
> sure, and I don't want to start installing a bunch of stuff that will
> clog up my directories and not even work.

There is a fork of setuptools called "distribute" that supports Python 3.

> What's the best way for me to install these two packages? I've also
> seen a reference to using setup.py...is that a separate package too,
> or is that something that comes with Python by default?

setup.py is a file that should be included at the top-level of the
.tar files you downloaded. Generally, to install something in that
manner, you would navigate to that top-level folder and run "python
setup.py install". If you have multiple Python versions installed and
want to install the package for a specific version, then you would use
that version of Python to run the setup.py file.
--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
On Wed, Mar 7, 2012 at 3:01 PM, Ian Kelly <ian.g.kelly@gmail.com> wrote:

> There is a fork of setuptools called "distribute" that supports Python 3.

Thanks, I guess I'll give this a try tonight!

> setup.py is a file that should be included at the top-level of the
> .tar files you downloaded.  Generally, to install something in that
> manner, you would navigate to that top-level folder and run "python
> setup.py install".  If you have multiple Python versions installed and
> want to install the package for a specific version, then you would use
> that version of Python to run the setup.py file.

The only files included in the .tar.gz file is a .tar file of the same
name. So I guess the setup option doesn't exist for these particular
packages. I'll try "distribute" tonight when I have some time to mess
with all of this.

So much work just to get a 3rd party module installed!
--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
On Wed, Mar 7, 2012 at 4:11 PM, John Salerno <johnjsal@gmail.com> wrote:
>
> On Wed, Mar 7, 2012 at 3:01 PM, Ian Kelly <ian.g.kelly@gmail.com> wrote:
>
> > There is a fork of setuptools called "distribute" that supports Python
> > 3.
>
> Thanks, I guess I'll give this a try tonight!
>
> > setup.py is a file that should be included at the top-level of the
> > .tar files you downloaded.  Generally, to install something in that
> > manner, you would navigate to that top-level folder and run "python
> > setup.py install".  If you have multiple Python versions installed and
> > want to install the package for a specific version, then you would use
> > that version of Python to run the setup.py file.
>
> The only files included in the .tar.gz file is a .tar file of the same
> name. So I guess the setup option doesn't exist for these particular
> packages. I'll try "distribute" tonight when I have some time to mess
> with all of this.
>
> So much work just to get a 3rd party module installed!
> --


It's because your extraction program is weird. Gzip is a compression
algorithm that operates on a single file. Tar is an archive format
that combines multiple files into a single file. When we say "extract
the .tar.gz", what we mean is both uncompress the tar file and then
extract everything out of that. A lot of programs will do that in one
step. If you look inside the tar file, you should find the setup.py.
--
http://mail.python.org/mailman/listinfo/python-list
RE: What's the best way to write this regular expression? [ In reply to ]
> The only files included in the .tar.gz file is a .tar file of the same
> name.

gz stands for gzip and is a form of compression (like rar/zip ).
tar stands for a tape archive. It is basically a box that holds the
files. So you need to "unzip" and then "open the box".

Normally programs like WinZip / WinRar / 7-zip will do both in one step
so you do not need to. Not sure what program you are using...

Ramit


Ramit Prasad | JPMorgan Chase Investment Bank | Currencies Technology
712 Main Street | Houston, TX 77002
work phone: 713 - 216 - 5423

--

This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.
--
http://mail.python.org/mailman/listinfo/python-list
Re: What's the best way to write this regular expression? [ In reply to ]
On Wed, Mar 7, 2012 at 2:11 PM, John Salerno <johnjsal@gmail.com> wrote:
> The only files included in the .tar.gz file is a .tar file of the same
> name. So I guess the setup option doesn't exist for these particular
> packages.

The setup.py file (as well as the other files) would be inside the
.tar file. Unlike a Windows zip file, which does both archival and
compression, Unix files are typically archived and compressed in two
separate steps: "tar" denotes the archival format, and "gz" denotes
the compression format. Some decompression programs are smart enough
to recognize the .tar file and automatically extract it when
decompressing. Others require you to decompress the .gz and extract
the .tar separately -- it sounds like yours is one of the latter.
--
http://mail.python.org/mailman/listinfo/python-list

1 2  View All