menu
  Home  ==>  papers  ==>  web  ==>  threaded_indy_news_reader   

Threaded Indy News Reader - Felix John COLIBRI.

  • abstract : a NewsReader which presents the articles sorted by thread and in a logical hierarchical way: who answers to whom. It uses the basic Indy NewsReader demo as a starting point
  • key words : NNTP reader - news reader - threaded newsreader
  • software used : Windows XP, Delphi 6, Indy 8.025
  • hardware used : Pentium 2.800Mhz, 512 M memory, 140 G hard disc
  • scope : Delphi 1 to 2006, Turbo Delphi for Windows, Kylix
  • level : Delphi developer
  • plan :


1 - Reading News

Newsgroup traffic is the biggest TCP/IP volume. For Delphi, newsgroups are both the best place to find answers to narrow problems, and a huge knowledge base.

You will find plenty of newsreaders around, even some free, and with sources. Having very specific needs, we built our own, which originated in a Compuserve BBS (Bulletin Board: the ancestor of the newsgroups) project. Recently, we had to read SSL (Secure Socket Layer) protected newsgroups, but we had not implemented the SSL layer. Since Indy includes the SSL part, we decided to trade our old Windows Sockets engine for and Indy based one.

We first looked at the Delphi demo project:

    C:\Program Files\Borland\Delphi6\Demos\Indy\NewsReader

which is a very fine demo. We used it as the starting point of this project.

Why didn't we use the Indy demo "as is". There are a couple of reasons

  • first articles are presented in a sequential manner. For daily overview of messages, that's fine. But for in-depth analysis, a presentation by thread (by message topic, or subject), the hierarchical presentation is obviously better suited
  • we have added additional functionalities on top of the basic newsreader
By the way, "thread" in this context means "topic", and has nothing to do with multitasking "threads".




2 - The Threaded Indy NewsReader

2.1 - The Basic Indy NewsReader

Just to recap, here are the steps to read news with Indy
   drop a tIdNNtp Client on your tForm

image

Set the Host (forums.borland.com) and Port (119) properties

   to fetch all the newsgroup from an Host, use:

IdNNTP1.Connect;
IdNNTP1.GetNewsgroupList(my_tStrings);

where my_tStrings is any tStrings descendent (tListBox.Items, tMemo1.Lines, my_tstringlist etc)

   you will get an entry for each news group, like

 
borland.public.delphi.jobs 22692 22481 y
borland.public.delphi.multimedia 7236 7046 y
borland.public.delphi.non-technical 607311 600469 y
borland.public.delphi.oleautomation 63539 62670 y
borland.public.delphi.oodesign 26836 26153 y
borland.public.delphi.opentoolsapi 12827 12641 y
borland.public.delphi.reporting-charting 40787 40504 y

   once you have selected one of the newsgroups (borland_public_delphi_oodesign for instance), you can request all the messages within some id range:

IdNNTP1.SelectGroup('borland.public.delphi.oodesign');
IdNNTP1.SendXOVER('26153-26836'my_message_tStrings);

and ItNNTP1 will send the XOVER command to the news server, with the message id range. The headers will be placed in the my_header_tStrings tStrings descendent

   here is a part of the answer (truncated in string length and message count):

 
26809 | Why has OODB failed? | "William Egge" <begge@eggcentric.com> | Mon, 4 Dec 2006
26810 | Re: Why has OODB failed? | Michael Baytalsky <mike@contextsoft.com> | Mon, 04 Dec 200
26811 | Re: Why has OODB failed? | Franz-Leo Chomse <franz-leo.chomse@samac.de> | Mon, 04 Dec 200
26812 | Re: Why has OODB failed? | "Wayne Niddery [TeamB]" <wniddery@chaffaci.on.ca> | Mon, 4 Dec
26813 | Re: SmallTalk | "Peter Morris [Droopy eyes software]" <pete@NO_droopyeyes_SPAM.
26814 | Re: Why has OODB failed? | Andreas Dorn <adornno1@web.de> | Mon, 04 Dec 200
26815 | Re: SmallTalk | "Jarle Stabell" <jarle@remove_stuff_dlogikk_spam_kills_email.co
26816 | Re: Why has OODB failed? | "Wayne Niddery [TeamB]" <wniddery@chaffaci.on.ca> Tue, 5 Dec
26817 | Re: Why has OODB failed? | "Bob Dawson" <RBDawson@prodigy.net> | Tue, 5 Dec 2006
26818 | Onion | "Peter Morris [Droopy eyes software]" <pete@NO_droopyeyes_SPAM.
26819 | TTreeView.ValidateSelection not virtual nor dynamic?! | Ricardo Villela Coppola <coppola@c
26820 | Re: SmallTalk | "Peter Morris [Droopy eyes software]" <pete@NO_droopyeyes_SPAM.
26821 | Re: SmallTalk | "Bart" <@> | Fri, 8 Dec 2006
26822 | Re: SmallTalk | "Peter Morris [Droopy eyes software]" <pete@NO_droopyeyes_SPAM.
26823 | Re: SmallTalk | "Bart" <@> | Fri, 8 Dec 2006
26824 | Re: Why has OODB failed? | "John Herbster" <herb-sci1_AT_sbcglobal.net> | Fri, 8 Dec 2006
26825 | Re: Why has OODB failed? | "Harley Pebley" <harley_pebley@idahotech.com> | 8 Dec 2006 16:4
26826 | Re: Why has OODB failed? | Michael Baytalsky <mike@contextsoft.com> | Sat, 09 Dec 200
26827 | Re: TTreeView.ValidateSelection not virtual nor dynamic?! | "Bob Dawson" <RBDawson@prodigy

We have truncated the length of each line, and replaced the Tab (Chr(9)) with a "|"

A full header line will look like

 
26809
  Why has OODB failed?
  "William Egge"
  <begge@eggcentric.com>
  Mon, 4 Dec 2006 08:47:08 -0500
  <457426d2$1@newsgroups.borland.com>
  2156
  39
  Xref: newsgroups.borland.com borland.public.delphi.oodesign:26809

   to donwload a single message (article), we drop a tIdMessage on your tForm:

image

  • request the message (Wayne Niddery's answer to William Egge in our case) using

    IdNNTP1.GetArticle(26812, ''IdMessage1);
    my_sting:=  TIdText(IdMessage1.MessageParts.Items[0]).Body.Text;

   if we display the body of the message in a tMemo, we would see (truncated):

 
William Egge wrote:
> I am curious why OODB has failed to take hold.  It can"t be because of
> needing compaitibility with other systems because an OODB should be
> able to provide a relational interface for legacy applications.

The model simply doesn"t work at the persistence level - one does not need 
to store *objects* (which include behaviour) one just needs to store the 
data of those objects.

Storing entire objects as some kind of blob is counter-productive because it 
cannot be effectively and efficiently queried, and it also inefficient since 
not all of attributes of an object *should* be stored - one of the nice 
things about objects is they can have derived, non-persistent attributes and 
relations.




So the tIdMessage is simply a component that analyzes the message's content. In our case, sending the basic Windows Socket ARTICLE request

my_nntp_socket.Send('ARTICLE 26812')

would have brought back the following text (truncated):

 
220 26812 <457449dc$1@newsgroups.borland.com> article retrieved - head and body follows
Reply-To: "Wayne Niddery [TeamB]" <wniddery@chaffaci.on.ca>
From: "Wayne Niddery [TeamB]" <wniddery@chaffaci.on.ca>
Newsgroups: borland.public.delphi.oodesign
References: <457426d2$1@newsgroups.borland.com>
Subject: Re: Why has OODB failed?
Date: Mon, 4 Dec 2006 11:17:08 -0500
Lines: 69
X-Priority: 3
X-MSMail-Priority: Normal
X-Newsreader: Microsoft Outlook Express 6.00.2900.2869
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2962
X-RFC2646: Format=Flowed; Response
NNTP-Posting-Host: 69.17.150.141
Message-ID: <457449dc$1@newsgroups.borland.com>
X-Trace: newsgroups.borland.com 1165248988 69.17.150.141 (4 Dec 2006 08:16:28 -0700)
X-Authenticated-User: wniddery
Path: newsgroups.borland.com!not-for-mail
Xref: newsgroups.borland.com borland.public.delphi.oodesign:26812

William Egge wrote:
> I am curious why OODB has failed to take hold.  It can"t be because of
> needing compaitibility with other systems because an OODB should be
> able to provide a relational interface for legacy applications.

The model simply doesn"t work at the persistence level - one does not need 
to store *objects* (which include behaviour) one just needs to store the 
data of those objects.

Storing entire objects as some kind of blob is counter-productive because it 
cannot be effectively and efficiently queried, and it also inefficient since 
not all of attributes of an object *should* be stored - one of the nice 
things about objects is they can have derived, non-persistent attributes and 
relations.



Therefore, using the Indy components has the following benefits:

  • we do not have to wait for and assemble the different TCP/IP packets
  • the status code (220) is analyzed for us
  • the intersting parts of the header (FROM, SUBJECT etc) are placed in different properties of the component
  • if the message is a multipart message, they are also separated in the tIdMessage.MessageParts.Items array. As I understand, Borland newsgroup Netiquette discourage using attachments (they are stored in a separate newsgroup), but they could be handy for some other Server


2.2 - Threaded NewsReader

Requesting messages by id (26.812 in our case) does not present a logical view of the conversation taking place:
  • several topic messages are intermingled
  • there is no indication to "who answers to whom"
For instance, using the original NTTP newsreader demo, for the "oodesign" group and the first "OODb" message, we would get:

image

You will notice that the "OODb" messages are mixed with the other messages ("Smalltalk" in our case)



To somehow represent the thread, we have to

  • sort the messages by thread
  • in each thread, display the hierarchical nature of the discussion
Here is an example of the hierarchy of the "OODb" thread:

 
Why has OODB failed?
  William Egge <begge@eggcentric.com> [26809]
    Michael Baytalsky <mike@contextsoft.com> [26810]
      John Herbster <herb-sci1_AT_sbcglobal.net> [26824]
        Michael Baytalsky <mike@contextsoft.com> [26826]
      Harley Pebley <harley_pebley@idahotech.com> [26825]
    Franz-Leo Chomse <franz-leo.chomse@samac.de> [26811]
    Wayne Niddery [TeamB] <wniddery@chaffaci.on.ca> [26812]
      Andreas Dorn <adornno1@web.de> [26814]
        Wayne Niddery [TeamB] <wniddery@chaffaci.on.ca> [26816]
          Bob Dawson <RBDawson@prodigy.net> [26817]

This is the purpose of the present project.

In fact, we presented already how to build the message tree structure in a detailed paper, with the algorithm we used, UML Class diagram and the UNIT which builds the tree. We are simply going to incorporate the CLASSes in the current project



2.3 - The Delphi Threaded NewsReader

We used the same basic organization as the Demo. Here are the modifications:
  • we removed the SSL parts, not required in our case
  • we simplified the newsgroup tListView, and added a simple optional "Delphi" filter
  • we removed the list of messages from a group
and we added the following parts
  • we replaced the list of message by a thread list
  • clicking on an Item of the thread list displays
    • either a tTreeView of the messages, and clicking on on of those fetches the content of the message
    • or we fetch all the messages of the thread and present them indented


2.4 - Mini User Guide

The easiest way to understand what our project is to look at a reading session:
   compile and run the p_threaded_news_reader project
   the project is displayed

image

   connect the reader to your server by clicking "connect_"
   all available newsgroups are displayed, along with the currently available message count:

image

Note that

  • since we checked the delphi_ tCheckbox, our project only presents the Delphi newsgroups. We have no need for any C++ or other Java stuff.
  • we also removed the "borland.public.delphi." prefix which does not add anything to the display
  • and we removed the empty groups
   to display the messages from a given newsgroup, click on the target newsgroup
   the project sends the "XOVER" NNTP command to fetch the message headers:

image

   after 5 seconds (ADSL, 512K), they are here, and our U_C_NEWS_THREAD_LIST UNIT orders them by thread, and presents them, along with their starting date (month/year in mm-yy format):

image

   to the message from a thread on by one, select the "article_" tab from the right tNoteBook, and click on one of the thread in the middle tTreeview. In our case, we chose "UML Class Diagram From Source"
   the tTreeView displays all message headers in an indented fashion:

image

   clicking on any message in the tTreeview requests the message content and displays it in a tMemo:

image

   to view all the message from a thread, select the "articleS_" tab from the right tNoteBook, and click on one of the thread in the middle tTreeview. In our case, we chose "UML Class Diagram From Source"
   the tMemo displays all the messages in an indented fashion. Here are the first message, and the start of Gerrit Beuze's answer:

image



Concerning the topic of the thread
  • Mr Remo Candeli is somehow hard with Model Maker. I still use it for our UML Trainings, when customers do not wish to use Together
  • Gerrit Beuze from ModelMaker Tools answers very nicely and even provides links to Model Maker UML analysis tools
  • to the list provided by Gerrit Beuze, I would also add the excellent, free, open source .PAS to UML Class diagram Ess-Model tool
All those comments have nothing to do with threaded newsreader, but each time I read a topic, I have a hard time to stop reading. Newsgroups are such an information gold mine...



2.5 - What's Next ?

We now have our basic message download tool, as well as a hierarchical article display. The true work will start from there.

Reading messages every morning is fine, but you cannot build a knowledge base unless you somehow save the text, and create either an indexing scheme, or a search engine.

Whenever I look for some information, I simply go to Tamarack an try to find some keyword that their Rubicon engine will use to locate relevant messages. The main reason I am using Tamarack, like so many of you, is that Tamarack has saved all the Delphi messages, and offers an Internet querying system to get the information out of this database.

What I would prefer is to download the archive, of, say, the last 5 years, and use our own Full Text Search engine which has a better requesting system (in my opinion). You have the full Pascal logic at your fingertips. To get tools other than ModelMaker which extract the UML Class diagram from source code we could ask:

"UML Class diagram" AND (extract OR "reverse engineering" OR "from source") AND NOT (ModelMaker OR "Model Maker")

I am not sure that Rubicon can handle such requests.

So why havn't we already implemented this machinery ? Well, our Delphi newsgroup archives are incomplete. When some customer projects dragged us into heavy programming, donwloading the news was often neglected. Maybe we will build an automatic tool which will do the downloading each morning, to avoid those gaps in our archives.

But this brings me to another pet idea of mine: why did Borland never offered the FULL archives as .ZIP files ? Are there some legal reasons (copyrights, maybe ?). Or disk space ? Or lack of interest from the developers ?

I mentioned it when I met some CodeGear executives, early October 2006, but have not received any answer so far. Maybe if many of you push for this .ZIP archive, and if the previous stuff has not yet been scrapped, we could get this fabulous knowledge base ? And if I can put my hands on the .ZIPs, I will publish the adapted search engine.




3 - Download the Sources

Here are the source code files:

The .ZIP file(s) contain:

  • the main program (.DPR, .DOF, .RES), the main form (.PAS, .DFM), and any other auxiliary form
  • any .TXT for parameters, samples, test data
  • all units (.PAS) for units
Those .ZIP
  • are self-contained: you will not need any other product (unless expressly mentioned).
  • for Delphi 6 projects, can be used from any folder (the pathes are RELATIVE)
  • will not modify your PC in any way beyond the path where you placed the .ZIP (no registry changes, no path creation etc).
To use the .ZIP:
  • create or select any folder of your choice
  • unzip the downloaded file
  • using Delphi, compile and execute
To remove the .ZIP simply delete the folder.

The Pascal code uses the Alsacian notation, which prefixes identifier by program area: K_onstant, T_ype, G_lobal, L_ocal, P_arametre, F_unction, C_lasse etc. This notation is presented in the Alsacian Notation paper.



As usual:

  • please tell us at fcolibri@felix-colibri.com if you found some errors, mistakes, bugs, broken links or had some problem downloading the file. Resulting corrections will be helpful for other readers
  • we welcome any comment, criticism, enhancement, other sources or reference suggestion. Just send an e-mail to fcolibri@felix-colibri.com.
  • or more simply, enter your (anonymous or with your e-mail if you want an answer) comments below and clic the "send" button
    Name :
    E-mail :
    Comments * :
     

  • and if you liked this article, talk about this site to your fellow developpers, add a link to your links page ou mention our articles in your blog or newsgroup posts when relevant. That's the way we operate: the more traffic and Google references we get, the more articles we will write.



4 - References

  • the basic Indy NewsReader demo is in the Delphi Demo folder
  • the construction of a thread tree is presented our the message tree structure paper, with the algorithm we used, UML Class diagram and the UNIT which builds the tree.
  • Indy components are on Delphi's Palette, but you can find additional information, a knowledgebase and additional demos at the Indy Project web site



5 - The author

Felix John COLIBRI works at the Pascal Institute. Starting with Pascal in 1979, he then became involved with Object Oriented Programming, Delphi, Sql, Tcp/Ip, Html, UML. Currently, he is mainly active in the area of custom software development (new projects, maintenance, audits, BDE migration, Delphi Xe_n migrations, refactoring), Delphi Consulting and Delph training. His web site features tutorials, technical papers about programming with full downloadable source code, and the description and calendar of forthcoming Delphi, FireBird, Tcp/IP, Web Services, OOP  /  UML, Design Patterns, Unit Testing training sessions.
Created: nov-04. Last updated: sep-16 - 107 articles, 228 .ZIP sources, 1174 figures
Copyright © Felix J. Colibri   http://www.felix-colibri.com 2004 - 2016. All rigths reserved
Back:    Home  Papers  Training  Delphi developments  Links  Download
the Pascal Institute

Felix J COLIBRI

+ Home
  + articles_with_sources
    + database
    + web_internet_sockets
      – tcp_ip_sniffer
      – socket_programming
      – socket_architecture
      – simple_web_server
      – simple_cgi_web_server
      – cgi_database_browser
      – whois
      – web_downloader
      – web_spider
      – rss_reader
      – news_message_tree
      – indy_news_reader
      – delphi_web_designer
      – intraweb_architecture
      – ajax_tutorial
      – bayesian_spam_filter
      + asp_net
      – mobile_pos_software
    + oop_components
    + uml_design_patterns
    + debug_and_test
    + graphic
    + controls
    + colibri_utilities
    + colibri_helpers
    + delphi
    + firemonkey
    + compilers
    + vcl
  + delphi_training
  + delphi_developments
  + sweet_home
  – download_zip_sources
  + links
Contacts
Site Map
– search :

RSS feed  
Blog