Google Doesn't Find My Forum's Content

[GregChapman]

⚫ Bigger Bullet

⚫ Bigger Bullet

But look through: https://www.w3schools.com/charsets/ref_utf_symbols.asp and the sections on Dingbats and Entities generally to appreciate all you could do!

[Gary Lewis]

Here's what my robots.txt file says, and it isn't excluding the production forum page.

[GregChapman]

Mmm! Perhaps it isn't Weebly after all.

But do check that Weebly doesn't link Nabble via its apps folder in some way - or something strange like that.

And maybe Google's robots.txt checker would confirm if that's the problem:
https://support.google.com/webmasters/answer/6062598?hl=en&ref_topic=6061961

[GregChapman]

Just tried it and http://forum.garysgaragemahal.com/ responds "ALLOWED".

[Gary Lewis]

The plot thickens!  

So, am I expecting the not-to-be-expected idea of Google finding the content of my forum when it is on the Nabble server?  In other words, just because it can crawl that page, maybe it can't crawl the Nabble server to which that page links?

[Gary Lewis]

Oh, and THANK YOU!

[GregChapman]

Gary Lewis wrote

The plot thickens!

Indeed!

As I said, I'm not an expert on robots.txt, but the balance of probability does seem to have swung back towards it being an interaction between Nabble and Weebly.

Perhaps the next step is to search Nabble Support for Weebly issues (most appear to using the embedding features of Weebly wrongly) and see if other Weebly sites are affected in the same way.

[GregChapman]

I'm struggling to find another weebly site to test.

They all seem to have been abandoned altogether or the Weebly sites that remain have abandoned Nabble. :-(

[Gary Lewis]

I'm at a loss.  Perhaps I have Nabble embedded improperly?

[GregChapman]

There's only one other idea I can offer to test if it might be on the Weebly side...

Try installing a site search. I use the free service from freefind.com for a number of my sites. The search results could go to a temporary page that's not linked to you main site.

Freefind allows me to set up the search both from my home page and from the Nabble forum home page and to include various exclusions. I use that to stop it picking up those Nabble pages which don't include the full bodies or topics.

If Freefind succeeds in finding your Nabble content then it should confirm that it is a Weebly issue. You could also see if it makes any difference if your site under a "Redirect to" or not. As you can get Freefind to crawl on demand, your site wouldn't need to be operating outside the Redirect for more than a few minutes. Whereas Google could take days to crawl you again.

[Gary Lewis]

Greg - Good idea.  I'll give freefind.com a try.  But I do have an embedded Google site-specific search on each page of the website as well as the Nabble search on the forum.  The Nabble search finds things like it should, but the Google search doesn't.  

So, I'll go look at freefind and potentially put it on each page in addition to the Google search.  That should tell us something..........

[GregChapman]

NO - IT'S DEFINITELY A NABBLE ISSUE or a change in the way Google crawls a site!

I've just re-visited my old post at:
http://support.nabble.com/SEO-and-embedded-forum-tp7590128p7590132.html
The results I claim there are not the results I get now.

Now all I get for: site:http://seahawk-forum.968426.n3.nabble.com hull
is  the home page of the site!

In short, for an embedded site all Google finds the home page of the forum.

If I try a search: site:nabble.com hull
I get innumerable results, but they are all from non-embedded Nabble sites.

[Gary Lewis]

I've added freefind to the footer such that it shows on every page.  And they've crawled the site, although they only got 337 of the roughly 400 pages.  However, they started on the home page which has the forum, so no problemo.

And, here's the results.  Obviously something is excluding the Nabble embedded code, although it isn't the robots.txt file as the one freefind linked to is the one I previously posted w/o any exclusions on the home/forum page.  And, predictably, freefind couldn't find anything on the forum but does find things from other pages on the site.

So, where is the exclusion coming from?

[Gary Lewis]

Well, the hunt is on!  If you have it too, and if freefind can't find it, then it isn't Google and it isn't Weebly.  

[GregChapman]

Hi Gary,

Not sure that I reach the same conclusion as you - but I'm not seeing all your log.

My approach is cruder. Go to:
http://www.seahawk17.org.uk/index.php
and enter "hull" and click the "Search Site" button.
The results that come up show many from the seahawk-forum.968426.n3.nabble.com forum so I know it's searching properly.

Whereas your log only appears to show only the home page of the forum is spidered. Have I got that wrong?

[GregChapman]

Hi Gary,

I had never looked at a Freefind log before. I decided to look at one of mine. I'm emailing you a copy so you can compare.

What I do notice is that the Nabble forum is excluded in yours. I'm guessing that will be because you haven't added the the forum as an additional starting point.

Because I have now realised that my site's forum is also not being spidered in the way it was in 2014, I think we can be sure it is related to being an embedded site.

As far as I'm aware I have done nothing in the last few months to stop it working. There's always someone posting on the forum who has failed to spot the answer they want somewhere on the site and I can't always remember where the snippet of info will be so I use the search fairly regularly. That's why I assumed that the issue was only affecting your site and never bothered to check.

[Gary Lewis]

Greg - I'm not sure where to start.  First, the forum is on my homepage, so I didn't think you needed to see more.  However, there are only two instances in the log of "Nabble", and they are shown below.

06:04:16  extracted url excluded     http://bullnose-enthusiasts.12971.n8.nabble.com/1983-vintage-advertisement-videos-tp1818.html   off site
06:06:30  extracted url excluded     http://bullnose-enthusiasts.12971.n8.nabble.com/template/NamlServlet.jtp?macro=user_nodes&user=77&cid=1511543874118-384   off site

But I see in your log that you have 9.087 instances of "Nabble".  Wow!

As for the starting points, I don't know how to do that, although I surely have a link to the Nabble server somewhere.  But, that would seem to only fix the Freefind searches, and my goal is Google.  So, what if I had a link to the Nabble server on my home page - would Google go search that?

[Gary Lewis]

Well, I added another starting point, as shown below, and things are looking up.  I found 57 instances of "Nabble" in the report, and the first search I did returned the correct result.

But I had tried placing that link to the Nabble server on the home page and then indexing it with Freefind and that didn't work.

Anyway, it is bed time and tomorrow is a busy day so I may not get back to this until tomorrow night.


Cheers,
Gary

[GregChapman]

Hi Gary,

Gary Lewis wrote

Well, I added another starting point, as shown below, and things are looking up.  I found 57 instances of "Nabble" in the report, and the first search I did returned the correct result.

I too find 57 pages on your site that include the term "nabble" in the content! :-)

But I had tried placing that link to the Nabble server on the home page and then indexing it with Freefind and that didn't work.

Not sure that I follow what you were doing by "placing that link to the Nabble server on the home page" and what you were expecting to achieve. However, for me, the important bit is that while Freefind clearly is capable of indexing your site, including pages in your embedded forum and Google does not! I used to assume that was something to do with Weebly or how you used it.

However, since yesterday, when I discovered that the forum on my SeaHawk site is no longer being indexed by Google, I have changed my mind and trying this search on GOOGLE, BING and YANDEX:

site:nabble.com seahawk hull

confirms the issue.

All Google finds is the home page of the forum - and plenty of other results but, importantly, all from non-embedded Nabble forums.

Whereas the other two engines all find plenty of results from my forum, with varying levels of priority for those showing "seahawk" within the URL.

So I think the answer is clear. Google has changed its algorithms (or Nabble has made a change that Google doesn't like) and it no longer is crawling any Nabble forum that is embedded.

Hopefully, Nabble can do something about that!

[Gary Lewis]

I'm searching for "My 4-door diesel project has a hopeless case of what Gary would call scope creep."  Google doesn't find it, but Bing does.

So, I fully agree.  Something is amiss between Google and Nabble for embedded forums.

I've done a lot a reading and haven't found a way to tell Google that my forum on Nabble's servers should be crawled.  But Bing does it, so.....