We build. You grow.

Get best community software here

Start a social network, a fan-site, an education project with oxwall - free opensource community software

Problems with site migration - Data Importer | Forum

Zoran Milojkovic
Zoran Milojkovic Nov 6 '13
Hello all,
I used Dataexporter on existing site and got its zip.
Then I copied SALT value into the ow_includes/config.php file to the OW_PASSWORD_SALT constant.
Then I installed Data Importer plugin on my new oxwall site, and uploaded this zip, started import.
First of all, all plugins are, after 1 hour of starting the import, still in the state 'wait import';
Second, there are no photos or any other user content in the exported zip file, inspite of many user uploaded files (511 users total) ;
I am wondering what did I miss, or what am I doing wrong.

Please advise,
Thanks,
Zoran
Zoran Milojkovic
Zoran Milojkovic Nov 6 '13
5 hours later there is still same message onthe DATA IMPORTER page:

Insert //snipped// into the ow_includes/config.php file to the OW_PASSWORD_SALT constant.

Import in progress. Please wait.


When I click on IMPORT STATUS button, I got this:

Import Plugins     Status
Virtual Gifts
Allow users to send private and public virtual gifts    wait import
Links
Web links sharing, Digg-style     wait import
Tiny Chat
Integration of popular, simple and easy to use chat with live audio and video     wait import
Groups
Simple groups within one site     wait import
Admin Panel    wait import
Mailbox
User mailbox for private messages     wait import
Blogs
User blogs with archives, tags, comments and rates     wait import
Base OW plugin Description     wait import
Contact Importer
Import Facebook, Google contacts     wait import
Birthdays
The list of upcoming birthdays for site users + widgets for index, dashboard and profiles    wait import
Activity notifications
Real-time and email notifications about site activity    wait import
Video
Allow users to embed videos with comments, rates, and tags    wait import
Newsfeed
Newsfeed with likes and comments, Facebook-style    wait import
Photo
Allow users to upload photos with tags, rates, and comments    wait import
Contact Us
"Contact us" page with the ability to choose departments (email addresses)    wait import
Events
Create public and private events within your community, let people RSVP and discuss    wait import
Friends
Friending functionality to be used across multiple features    wait import
Forum
Simple discussion boards for users    wait import


I also found in documentation that configuring cron job like this:
wget http://www.yousiteurl.com/run.php
won't make import.
But, I have my cron configured like this:
lynx http://url of the site/ow_cron/run.php


Please advise what I am doing wrong.
The Forum post is edited by Zoran Milojkovic Nov 6 '13
Alia Team
Alia Nov 6 '13
Zoran,

1. Have you checked  what is you max. execution time. Should be at least 30 mins.
2. Have you checked "ow_dataimporter_import" table in your database? Where does it get stuck?
Can you give  screenshot of this table?
3. You will need to check whether Cron is indeed working on your new Oxwall installation.
Easiest way to test this is by inviting somebody using Contact Importer plugin ( invite by email, don't test with FB/Google invites). If email arrives- your Cron is working.

Zoran Milojkovic
Zoran Milojkovic Nov 7 '13
Hi Aliia,
Thanks for a reply. Here are requested data:
1. php parameters
Resource Limits     max_execution_time         30
Resource Limits     max_input_time         60
Resource Limits     memory_limit         64M
...
File Uploads     upload_max_filesize     Maximum allowed size for uploaded files.     1024M

2. There was no ow_dataimporter_import in the database until I did change from lynx to curl. Please find attached screenshot of the table.

3. I get every minute mail message about cron job, I also checked your proposed way, did not work until I configured curl to call the cron instead of lynx, so it works now.

What is also noticeable is that, inspite of correct value in config.php file and in database, there is still a warning on the plugin page about salt value.

Please advise,
Zoran
The Forum post is edited by Zoran Milojkovic Nov 7 '13
Attachments:
  dataimporter_table.jpg (161.1Kb)
Skalfa LLC Partner
Skalfa LLC Nov 11 '13
Zoran,  Data Importer/Exporter are our plugins.

Thank you for your answers.
Seems like the issue is with Cron.
You will need to set your Cron locally. Your command should look like:

/usr/local/bin/php -f /home/username/public_html/ow_cron/run.php

To find out what is your path follow instructions given here: http://docs.oxwall.org/install:cron ; ( section " Run Cron Locally").

Once you change your command, make sure that it is working ( try inviting member).
Reinstall "Data Importer" plugin and try to import your site again.
The Forum post is edited by Skalfa LLC Nov 11 '13
Zoran Milojkovic
Zoran Milojkovic Nov 11 '13
Thank you Skalfa for your reply.
The only working way of invoking cron job on my hosting is this (I tried all suggested in Oxwall documentation):
curl http://sajbernetiks.rs/kolo/ow_cron/run.php
It works if sending invite by email (inviting a member) is sufficient for test.
Reinstalled Data importer.
Tried to import data again.
No change, same report.

What also worries me (this is another backup issue) is that Data exporter did not collect any of user files (eg. profile pics etc.).

Please advise.
Thanks,
Zoran
Skalfa LLC Partner
Skalfa LLC Nov 12 '13
Zoran, Data Importer requires Cron to be executed via php ( locally), since this way it bypasses limits associated with webservers.

Executing cron  via curl or wget will work for regular routine ( ex. sending out emails), but not for data import which needs some time to return the response.

>>The only working way of invoking cron job on my hosting is this

You mean, your hosting company restricts running cron via php?
OR
You just were not able to find out the right path to your php and therefore were not able to make the cron work properly?

Answering your concern about Data Exporter.
I guess you have just looked into the .zip file you got after exporting. It will not contain any folders/files with users' content ( avatars, photos, etc.). Migration of the content happens within the database structure ( via sql queries). So don't worry that you don't have your user files at this stage.

The Forum post is edited by Skalfa LLC Nov 12 '13
Zoran Milojkovic
Zoran Milojkovic Nov 12 '13
Thank you Skalfa,
Finally got cron right. Maybe I was no patient enough to test friend invite function, and rushed to conclusion that local execution is not working.
Anyway, I got some import, Import page of Data Importer is now empty, but database table is not, let's say, right, some plugins are stopped, some are started inspite of several hours past since beginning of the import. It is very strange that I got Server 500 Error page for at least an hour after starting an Import, so I was unable to check the progress of the import untill all was over.
After examination of imported data, I can say that user avatars are transferred, etc... unlike: photos, videos, forums and blogs. I checked all these .sql files in zip file - there are existing and non-empty.

What is also weird, I got Under maintenance page after logout.

Please advise, how can I restore missing data, I mean data that exist in backup but are not transferred into new database?

Thanks again,
Zoran
The Forum post is edited by Zoran Milojkovic Nov 12 '13
Attachments:
  dataimporter_table - 1.jpg (196.91Kb)
Skalfa LLC Partner
Skalfa LLC Nov 13 '13
Zoran,

>>After examination of imported data, I can say that user avatars are transferred, etc... unlike: photos, videos, forums and blogs.

Import was not finished, since it failed on "Mailbox" plugin ( that is why data of all plugins bellow "Mailbox" on your screen shot was not transferred to your new site).
Can you provide the full text of the error next to "Mailbox" row?

500phtml error means that something went wrong. To see the real error behind 500phtml one, you just need to enable "DEBUG" mode in config.php. Most likely this was something related to mailbox.

Site always goes to maintenance mode during the import so that process are not interrupted. If import finishes successfully  system disables maintenance mode automatically.

If your site is still under maintenance, then import failed.
Alia Team
Alia Nov 13 '13
Topic was moved from Bug reports and troubleshooting.
Zoran Milojkovic
Zoran Milojkovic Nov 13 '13

Quote from Skalfa LLC Can you provide the full text of the error next to "Mailbox" row?

A small portion of this error box, since the rest is sql text of the import:

0 snipped dir/ow_plugins/data_importer/bol/import_service.php 269
SQL import failed! SQL: ( '100','46','1333299259','1','58','
The Forum post is edited by Zoran Milojkovic Nov 19 '13
Zoran Milojkovic
Zoran Milojkovic Nov 13 '13
Thank you Skalfa,
Quote from Skalfa LLC 500phtml error means that something went wrong. To see the real error behind 500phtml one, you just need to enable "DEBUG" mode in config.php. Most likely this was something related to mailbox.
After I enable DEBUG mode, should I try to import all backup again or...?
Zoran Milojkovic
Zoran Milojkovic Nov 19 '13
Hello Skalfa,
Since there was no reply from you almpst for a week, I decided to turno on DEBUG mode and retry with import.
There was no progress in import, it broke in the same point and error message during
"

OW Debug - Exception

Message: Can't find master page `'.html_document.'` ! File: snipped dir/ow_core/theme_manager.php Line: 229 Trace: #0 site dir snipped/ow_core/html_document.php(580): OW_ThemeManager->getMasterPageTemplate('html_document') #1 site dir snipped/ow_core/response.php(184): OW_HtmlDocument->render() #2 site dir snipped/ow_core/application.php(429): OW_Response->respond() #3 site dir snipped/index.php(83): OW_Application->returnResponse() #4 {main} Type: InvalidArgumentException"
and it is shown on every page refresh during approxx 30 minutes after import start.
Same error in the Mailbox row in the table.

Should I try to perform manual tables import (if it is possible)?

Please advise.
Thanks,
Zoran
The Forum post is edited by Zoran Milojkovic Nov 19 '13
Skalfa LLC Partner
Skalfa LLC Dec 5 '13
Zoran, is this issue still relevant?
Zoran Milojkovic
Zoran Milojkovic Dec 9 '13
Yes, Skalfa, it is. No matter of how much we expand time out for php (max exec time set up to 60 minutes) it fails on same point.
I even tried to change the db code, e.g to do inserts more often in the query, but it broke the procedure (error was reported earlier in the execution of some previous table insert).

I have no idea what to try next.

Maybe I will disregard all the attached files and manually import all missing tables in the database...

Please advise,
Zoran
Zoran Milojkovic
Zoran Milojkovic Dec 22 '13
Another two weeks passes with no advise at all from the vendor... I tried to improve cron settings ant to more increase php timeou, but now the site is offline for more than 28 hours with this page:

OW Debug - Exception

Message: Can't find master page `'.html_document.'` ! File:
(snipped)/koloproba/ow_core/theme_manager.php Line: 229 Trace: #0 (snipped)/koloproba/ow_core/html_document.php(580): OW_ThemeManager->getMasterPageTemplate('html_document') #1 (snipped)/koloproba/ow_core/response.php(184): OW_HtmlDocument->render() #2 (snipped)/koloproba/ow_core/application.php(429): OW_Response->respond() #3 (snipped)/koloproba/index.php(83): OW_Application->returnResponse() #4 {main} Type: InvalidArgumentException
Please advise ASAP!
The Forum post is edited by Zoran Milojkovic Dec 22 '13
Zoran Milojkovic
Zoran Milojkovic Dec 25 '13
Well, I can say that after two and a half months I managed manualy, completely on my own, to move wall.fm site on new one, price payed was in missing attached files. Moving via plugins was unfeasable due to Data import plugin breaking during importing.

Anyway, thank you Skalfa and your team for making this great social networking software. Please make improvements in Data importer.

Regards,
Zoran
You do not have permission to reply this topic