Bei der Gruppe, für die Sie eine Mitteilung verfassen, handelt es sich um eine Usenet-Gruppe. Wenn Sie in dieser Gruppe Nachrichten posten, ist Ihre E-Mail-Adresse für jeden im Internet sichtbar
Presently it has more than 50,000 game entries, along with associated
publisher, artists, designers, and so forth. In other words--a
massive amount of information.
Everything is structured fairly well with proper model associations, I
think, so it's a fairly trivial task to pull the data out of the MySQL
database. The issue is that the resulting data array is huge...and I'm
running out of memory.
I can sometimes handle this by pulling 1000 records at a time, say,
but then I run into my next issue. To actually do all the reading and
writing to XML (via the SimpleXML class), I'd say it takes around 4 or
5 minutes. On my local environment, I can change the maximum
execution time to be limitless, so it's not a huge issue. But on my
Web host, though, that's not such an easy task--I'm not even sure it
can be done.
So, it gets to the main question: What's the best way to handle
processing of tens of thousands of records, reading them from a MySQL
database then writing them out via XML? (Incidentally, the XML file
that gets written is some 800MB in size!)
I have similar problem to your with massive mailing list where every
mail have to be send with some personalized data.
The solutions which works great was to write an small javascript and
using ajax calls php to send mails divided in portions (20 mails per
ajax call) so i've never execute php script more than 5-6 seconds (In
your case you can export your xml data in parts and append it to file
every ajax call).
I hope this give you some clue how to try to solve your problem.
Best Regards,
Quess
On Nov 20, 9:51 pm, carl67lp <ccander...@gmail.com> wrote:
> Presently it has more than 50,000 game entries, along with associated
> publisher, artists, designers, and so forth. In other words--a
> massive amount of information.
> Everything is structured fairly well with proper model associations, I
> think, so it's a fairly trivial task to pull the data out of the MySQL
> database. The issue is that the resulting data array is huge...and I'm
> running out of memory.
> I can sometimes handle this by pulling 1000 records at a time, say,
> but then I run into my next issue. To actually do all the reading and
> writing to XML (via the SimpleXML class), I'd say it takes around 4 or
> 5 minutes. On my local environment, I can change the maximum
> execution time to be limitless, so it's not a huge issue. But on my
> Web host, though, that's not such an easy task--I'm not even sure it
> can be done.
> So, it gets to the main question: What's the best way to handle
> processing of tens of thousands of records, reading them from a MySQL
> database then writing them out via XML? (Incidentally, the XML file
> that gets written is some 800MB in size!)
> I have similar problem to your with massive mailing list where every
> mail have to be send with some personalized data.
> The solutions which works great was to write an small javascript and
> using ajax calls php to send mails divided in portions (20 mails per
> ajax call) so i've never execute php script more than 5-6 seconds (In
> your case you can export your xml data in parts and append it to file
> every ajax call).
> I hope this give you some clue how to try to solve your problem.
> Best Regards,
> Quess
> On Nov 20, 9:51 pm, carl67lp <ccander...@gmail.com> wrote:
> > I have a database of board games; its data is based on the XML feed
> > from:
> > Presently it has more than 50,000 game entries, along with associated
> > publisher, artists, designers, and so forth. In other words--a
> > massive amount of information.
> > Everything is structured fairly well with proper model associations, I
> > think, so it's a fairly trivial task to pull the data out of the MySQL
> > database. The issue is that the resulting data array is huge...and I'm
> > running out of memory.
> > I can sometimes handle this by pulling 1000 records at a time, say,
> > but then I run into my next issue. To actually do all the reading and
> > writing to XML (via the SimpleXML class), I'd say it takes around 4 or
> > 5 minutes. On my local environment, I can change the maximum
> > execution time to be limitless, so it's not a huge issue. But on my
> > Web host, though, that's not such an easy task--I'm not even sure it
> > can be done.
> > So, it gets to the main question: What's the best way to handle
> > processing of tens of thousands of records, reading them from a MySQL
> > database then writing them out via XML? (Incidentally, the XML file
> > that gets written is some 800MB in size!)
|
