Fetch Posts From HIVE Blockchain
| Please Select a Licence from the LICENCE_HEADERS page |
And place at top of your page |
If no Licence is Selected/Appended, Default will be CC0 Default Licence IF there is no Licence placed below this notice!
When you edit this page, you agree to release your contribution under the CC0 Licence LICENCE:
More information about the cc0 licence can be found here: You can copy, modify, distribute and perform the work, even for commercial purposes, all without asking permission. Licence: Statement of Purpose The laws of most jurisdictions throughout the world automatically confer exclusive Copyright and Related Rights (defined below) upon the creator and subsequent owner(s) (each and all, an "owner") of an original work of authorship and/or a database (each, a "Work"). Certain owners wish to permanently relinquish those rights to a Work for the purpose of contributing to a commons of creative, cultural and scientific works ("Commons") that the public can reliably and without fear of later claims of infringement build upon, modify, incorporate in other works, reuse and redistribute as freely as possible in any form whatsoever and for any purposes, including without limitation commercial purposes. These owners may contribute to the Commons to promote the ideal of a free culture and the further production of creative, cultural and scientific works, or to gain reputation or greater distribution for their Work in part through the use and efforts of others. For these and/or other purposes and motivations, and without any expectation of additional consideration or compensation, the person associating CC0 with a Work (the "Affirmer"), to the extent that he or she is an owner of Copyright and Related Rights in the Work, voluntarily elects to apply CC0 to the Work and publicly distribute the Work under its terms, with knowledge of his or her Copyright and Related Rights in the Work and the meaning and intended legal effect of CC0 on those rights. 1. Copyright and Related Rights. A Work made available under CC0 may be protected by copyright and related or neighboring rights ("Copyright and Related Rights"). Copyright and Related Rights include, but are not limited to, the following: the right to reproduce, adapt, distribute, perform, display, communicate, and translate a Work; moral rights retained by the original author(s) and/or performer(s); publicity and privacy rights pertaining to a person's image or likeness depicted in a Work; rights protecting against unfair competition in regards to a Work, subject to the limitations in paragraph 4(a), below; rights protecting the extraction, dissemination, use and reuse of data in a Work; database rights (such as those arising under Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, and under any national implementation thereof, including any amended or successor version of such directive); and other similar, equivalent or corresponding rights throughout the world based on applicable law or treaty, and any national implementations thereof. 2. Waiver. To the greatest extent permitted by, but not in contravention of, applicable law, Affirmer hereby overtly, fully, permanently, irrevocably and unconditionally waives, abandons, and surrenders all of Affirmer's Copyright and Related Rights and associated claims and causes of action, whether now known or unknown (including existing as well as future claims and causes of action), in the Work (i) in all territories worldwide, (ii) for the maximum duration provided by applicable law or treaty (including future time extensions), (iii) in any current or future medium and for any number of copies, and (iv) for any purpose whatsoever, including without limitation commercial, advertising or promotional purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each member of the public at large and to the detriment of Affirmer's heirs and successors, fully intending that such Waiver shall not be subject to revocation, rescission, cancellation, termination, or any other legal or equitable action to disrupt the quiet enjoyment of the Work by the public as contemplated by Affirmer's express Statement of Purpose. 3. Public License Fallback. Should any part of the Waiver for any reason be judged legally invalid or ineffective under applicable law, then the Waiver shall be preserved to the maximum extent permitted taking into account Affirmer's express Statement of Purpose. In addition, to the extent the Waiver is so judged Affirmer hereby grants to each affected person a royalty-free, non transferable, non sublicensable, non exclusive, irrevocable and unconditional license to exercise Affirmer's Copyright and Related Rights in the Work (i) in all territories worldwide, (ii) for the maximum duration provided by applicable law or treaty (including future time extensions), (iii) in any current or future medium and for any number of copies, and (iv) for any purpose whatsoever, including without limitation commercial, advertising or promotional purposes (the "License"). The License shall be deemed effective as of the date CC0 was applied by Affirmer to the Work. Should any part of the License for any reason be judged legally invalid or ineffective under applicable law, such partial invalidity or ineffectiveness shall not invalidate the remainder of the License, and in such case Affirmer hereby affirms that he or she will not (i) exercise any of his or her remaining Copyright and Related Rights in the Work or (ii) assert any associated claims and causes of action with respect to the Work, in either case contrary to Affirmer's express Statement of Purpose. 4. Limitations and Disclaimers. No trademark or patent rights held by Affirmer are waived, abandoned, surrendered, licensed or otherwise affected by this document. Affirmer offers the Work as-is and makes no representations or warranties of any kind concerning the Work, express, implied, statutory or otherwise, including without limitation warranties of title, merchantability, fitness for a particular purpose, non infringement, or the absence of latent or other defects, accuracy, or the present or absence of errors, whether or not discoverable, all to the greatest extent permissible under applicable law. Affirmer disclaims responsibility for clearing rights of other persons that may apply to the Work or any use thereof, including without limitation any person's Copyright and Related Rights in the Work. Further, Affirmer disclaims responsibility for obtaining any necessary consents, permissions or other rights required for any use of the Work. Affirmer understands and acknowledges that Creative Commons is not a party to this document and has no duty or obligation with respect to this CC0 or use of the Work. |
- Fetching a syntax_wiki post from HIVE
Container Setup
- create container
lxc launch ubuntu:24.04 hive
- Login to container
lxc exec hive bash
- switch to user ubuntu
su - ubuntu
Update System and Install Dependencies
- Update package lists
sudo apt update && sudo apt upgrade -y
- Install essential packages
sudo apt install -y python3 python3-pip python3-venv git build-essential
- Install development libraries needed for compilation
sudo apt install -y libssl-dev libffi-dev python3-dev
- Verify Python installation
python3 --version
pip3 --version
Create Virtual Environment (Recommended)
- Create a virtual environment
python3 -m venv hive_beem_env
- Activate the virtual environment
source hive_beem_env/bin/activate
- Verify you're in the virtual environment (should show the path)
(hive_beem_env) ubuntu@hive:~$
Install Beem
- Install beem from PyPI
- NOTE: Beem is installed in Virtual Environment to avoid
error: externally-managed-environment
pip install beem
Script to Fetch Post from HIVE
$EDITOR hive-fetch.py
- WARNING THIS SCRIPT IS EARLY CONCEPT DRAFT BUGGY!
#!/usr/bin/env python3
"""
Hive Post Fetcher using Beem
Fetches raw markdown content from Hive blockchain posts
"""
import argparse
import json
import time
import xml.etree.ElementTree as ET
import xml.dom.minidom as minidom
import re
from datetime import datetime
from urllib.parse import urlparse
from beem import Hive
from beem.comment import Comment
from beem.account import Account
class HivePostFetcher:
def __init__(self, nodes=None):
if nodes is None:
nodes = [
"https://api.hive.blog",
"https://rpc.ecency.com",
"https://api.deathwing.me",
"https://fin.hive.3speak.co"
]
self.hive = Hive(node=nodes)
def parse_hive_url(self, url):
"""Parse Hive blog URL to extract author and permlink"""
try:
# Remove trailing slash if present
url = url.rstrip('/')
# Handle different URL formats:
# https://hive.blog/completenoobs/@completenoobs/completenoobs-genesis
# https://hive.blog/@completenoobs/completenoobs-genesis
# https://peakd.com/@completenoobs/completenoobs-genesis
# Extract the path part
parsed = urlparse(url)
path = parsed.path
# Split and filter empty parts
parts = [p for p in path.split('/') if p]
if len(parts) >= 2:
# Find the author (starts with @)
author_part = None
permlink_part = None
for i, part in enumerate(parts):
if part.startswith('@'):
author_part = part[1:] # Remove @
if i + 1 < len(parts):
permlink_part = parts[i + 1]
break
if author_part and permlink_part:
return f"@{author_part}/{permlink_part}"
return None
except Exception as e:
print(f"Error parsing URL: {e}")
return None
def fetch_single_post(self, authorperm_or_url):
"""Fetch a single post by author/permlink or URL"""
try:
# Check if input is a URL
if authorperm_or_url.startswith('http'):
authorperm = self.parse_hive_url(authorperm_or_url)
if not authorperm:
return {'success': False, 'error': 'Could not parse URL format'}
print(f"Parsed URL to: {authorperm}")
else:
authorperm = authorperm_or_url
post = Comment(authorperm, blockchain_instance=self.hive)
return {
'success': True,
'data': {
'title': post.get('title', 'No Title'),
'author': post['author'],
'permlink': post['permlink'],
'authorperm': post['authorperm'],
'created': str(post['created']),
'body': post['body'],
'url': f"https://hive.blog/@{post['author']}/{post['permlink']}",
'tags': post.get('json_metadata', {}).get('tags', []) if post.get('json_metadata') else [],
'category': post.get('category', ''),
'votes': len(post.get('active_votes', [])),
'payout': str(post.get('pending_payout_value', '0.000 HBD'))
}
}
except Exception as e:
return {'success': False, 'error': str(e)}
def fetch_user_posts(self, username, limit=50):
"""Fetch all posts by a user using get_account_posts"""
try:
account = Account(username, blockchain_instance=self.hive)
posts = []
# Use get_account_posts method (correct approach)
for post in account.get_account_posts(sort="posts", limit=limit):
try:
posts.append({
'title': post.get('title', 'No Title'),
'author': post['author'],
'permlink': post['permlink'],
'authorperm': post['authorperm'],
'created': str(post.get('created', '')),
'body': post['body'],
'url': f"https://hive.blog/@{post['author']}/{post['permlink']}",
'tags': post.get('json_metadata', {}).get('tags', []) if post.get('json_metadata') else [],
'category': post.get('category', ''),
'votes': len(post.get('active_votes', [])),
'payout': str(post.get('pending_payout_value', '0.000 HBD'))
})
time.sleep(0.3) # Be nice to the API
except Exception as e:
print(f"Error processing post: {e}")
continue
return {'success': True, 'data': posts}
except Exception as e:
return {'success': False, 'error': str(e)}
def save_as_json(self, posts, filename):
"""Save posts as JSON"""
with open(filename, 'w', encoding='utf-8') as f:
json.dump(posts, f, indent=2, ensure_ascii=False)
print(f"✅ Saved {len(posts)} posts to {filename}")
def save_as_xml(self, posts, filename):
"""Save posts as XML"""
root = ET.Element("hive_posts")
root.set("generated", datetime.now().isoformat())
root.set("count", str(len(posts)))
for post in posts:
post_elem = ET.SubElement(root, "post")
# Add post attributes
for key, value in post.items():
if key == 'tags' and isinstance(value, list):
tags_elem = ET.SubElement(post_elem, "tags")
for tag in value:
tag_elem = ET.SubElement(tags_elem, "tag")
tag_elem.text = str(tag)
elif key == 'body':
# Handle body content with CDATA
body_elem = ET.SubElement(post_elem, "body")
body_elem.text = str(value)
else:
elem = ET.SubElement(post_elem, key)
elem.text = str(value)
# Pretty print XML
rough_string = ET.tostring(root, encoding='unicode')
reparsed = minidom.parseString(rough_string)
pretty_xml = reparsed.toprettyxml(indent=" ")
with open(filename, 'w', encoding='utf-8') as f:
f.write(pretty_xml)
print(f"✅ Saved {len(posts)} posts to {filename}")
def save_as_wiki(self, posts, filename):
"""Save posts as MediaWiki syntax"""
with open(filename, 'w', encoding='utf-8') as f:
f.write("= Hive Posts Export =\n\n")
f.write(f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n")
f.write(f"Total Posts: {len(posts)}\n\n")
for i, post in enumerate(posts, 1):
# Post header
f.write(f"== Post {i}: {post['title']} ==\n\n")
# Post metadata table
f.write("{| class=\"wikitable\"\n")
f.write("|-\n")
f.write("! Property !! Value\n")
f.write("|-\n")
f.write(f"| Author || @{post['author']}\n")
f.write("|-\n")
f.write(f"| Created || {post['created']}\n")
f.write("|-\n")
f.write(f"| URL || [{post['url']} View on Hive]\n")
f.write("|-\n")
f.write(f"| AuthorPerm || {post['authorperm']}\n")
f.write("|-\n")
f.write(f"| Category || {post.get('category', 'N/A')}\n")
f.write("|-\n")
f.write(f"| Votes || {post.get('votes', 0)}\n")
f.write("|-\n")
f.write(f"| Payout || {post.get('payout', 'N/A')}\n")
# Tags
if post.get('tags'):
tags_str = ', '.join([f"[[{tag}]]" for tag in post['tags']])
f.write("|-\n")
f.write(f"| Tags || {tags_str}\n")
f.write("|}\n\n")
# Post content
f.write("=== Content ===\n\n")
# Convert markdown to wiki syntax (basic conversion)
wiki_content = self.markdown_to_wiki(post['body'])
f.write(wiki_content)
f.write("\n\n")
f.write("----\n\n") # Horizontal rule between posts
print(f"✅ Saved {len(posts)} posts to {filename}")
def markdown_to_wiki(self, markdown_text):
"""Basic conversion from Markdown to MediaWiki syntax"""
wiki_text = markdown_text
# Headers
wiki_text = re.sub(r'^# (.*)', r'= \1 =', wiki_text, flags=re.MULTILINE)
wiki_text = re.sub(r'^## (.*)', r'== \1 ==', wiki_text, flags=re.MULTILINE)
wiki_text = re.sub(r'^### (.*)', r'=== \1 ===', wiki_text, flags=re.MULTILINE)
wiki_text = re.sub(r'^#### (.*)', r'==== \1 ====', wiki_text, flags=re.MULTILINE)
# Bold and italic
wiki_text = re.sub(r'\*\*(.*?)\*\*', r"'''\1'''", wiki_text) # Bold
wiki_text = re.sub(r'\*(.*?)\*', r"''\1''", wiki_text) # Italic
# Links
wiki_text = re.sub(r'\[([^\]]+)\]\(([^)]+)\)', r'[\2 \1]', wiki_text)
# Code blocks
wiki_text = re.sub(r'```(\w*)\n(.*?)```', r'<syntaxhighlight lang="\1">\n\2</syntaxhighlight>', wiki_text, flags=re.DOTALL)
wiki_text = re.sub(r'`([^`]+)`', r'<code>\1</code>', wiki_text)
# Lists (basic conversion)
wiki_text = re.sub(r'^- (.*)', r'* \1', wiki_text, flags=re.MULTILINE)
wiki_text = re.sub(r'^\d+\. (.*)', r'# \1', wiki_text, flags=re.MULTILINE)
return wiki_text
def main():
parser = argparse.ArgumentParser(description='Fetch posts from Hive blockchain')
parser.add_argument('--user', type=str, help='Username to fetch posts from')
parser.add_argument('--post', type=str, help='Single post to fetch (format: @author/permlink or full URL)')
parser.add_argument('--url', type=str, help='Hive blog URL to fetch (e.g., https://hive.blog/@user/post)')
parser.add_argument('--limit', type=int, default=20, help='Limit number of posts to fetch')
parser.add_argument('--output', type=str, help='Output file name (extension determines format)')
parser.add_argument('--format', type=str, choices=['json', 'xml', 'wiki', 'txt'],
default='json', help='Output format (json, xml, wiki, txt)')
args = parser.parse_args()
fetcher = HivePostFetcher()
# Handle single post (either --post or --url)
single_post_input = args.post or args.url
if single_post_input:
result = fetcher.fetch_single_post(single_post_input)
if result['success']:
post_data = result['data']
print(f"Title: {post_data['title']}")
print(f"Author: @{post_data['author']}")
print(f"Created: {post_data['created']}")
print(f"URL: {post_data['url']}")
print(f"Category: {post_data['category']}")
print(f"Tags: {', '.join(post_data['tags'])}")
print(f"Votes: {post_data['votes']}")
print(f"Payout: {post_data['payout']}")
print("\n" + "="*60 + "\n")
print(post_data['body'])
# Save single post if output specified
if args.output:
posts_list = [post_data]
output_format = args.format
# Auto-detect format from filename extension if not specified
if '.' in args.output:
ext = args.output.split('.')[-1].lower()
if ext in ['json', 'xml', 'wiki', 'txt']:
output_format = ext
if output_format == 'json':
fetcher.save_as_json(posts_list, args.output)
elif output_format == 'xml':
fetcher.save_as_xml(posts_list, args.output)
elif output_format == 'wiki':
fetcher.save_as_wiki(posts_list, args.output)
else: # txt format
with open(args.output, 'w', encoding='utf-8') as f:
f.write(f"Title: {post_data['title']}\n")
f.write(f"Author: @{post_data['author']}\n")
f.write(f"Created: {post_data['created']}\n")
f.write(f"URL: {post_data['url']}\n")
f.write(f"AuthorPerm: {post_data['authorperm']}\n")
f.write("="*60 + "\n\n")
f.write(post_data['body'])
print(f"✅ Saved to {args.output}")
else:
print(f"Error: {result['error']}")
elif args.user:
result = fetcher.fetch_user_posts(args.user, args.limit)
if result['success']:
posts = result['data']
print(f"Fetched {len(posts)} posts for @{args.user}")
if args.output:
output_format = args.format
# Auto-detect format from filename extension
if '.' in args.output:
ext = args.output.split('.')[-1].lower()
if ext in ['json', 'xml', 'wiki', 'txt']:
output_format = ext
if output_format == 'json':
fetcher.save_as_json(posts, args.output)
elif output_format == 'xml':
fetcher.save_as_xml(posts, args.output)
elif output_format == 'wiki':
fetcher.save_as_wiki(posts, args.output)
else: # txt format
with open(args.output, 'w', encoding='utf-8') as f:
for i, post in enumerate(posts, 1):
f.write(f"{'='*60}\n")
f.write(f"POST {i}: {post['title']}\n")
f.write(f"Author: @{post['author']}\n")
f.write(f"Created: {post['created']}\n")
f.write(f"URL: {post['url']}\n")
f.write(f"AuthorPerm: {post['authorperm']}\n")
f.write(f"Tags: {', '.join(post['tags'])}\n")
f.write(f"{'='*60}\n\n")
f.write(post['body'])
f.write(f"\n\n{'='*60}\n\n")
print(f"✅ Saved to {args.output}")
else:
for i, post in enumerate(posts, 1):
print(f"\n{i}. {post['title']}")
print(f" Created: {post['created']}")
print(f" URL: {post['url']}")
print(f" Tags: {', '.join(post['tags'])}")
else:
print(f"Error: {result['error']}")
else:
parser.print_help()
print("\nExamples:")
print(" # Fetch user posts as JSON")
print(" python hive_fetcher.py --user completenoobs --limit 10 --output posts.json")
print("")
print(" # Fetch user posts as XML")
print(" python hive_fetcher.py --user completenoobs --output posts.xml")
print("")
print(" # Fetch user posts as Wiki syntax")
print(" python hive_fetcher.py --user completenoobs --output posts.wiki")
print("")
print(" # Fetch specific post by URL")
print(" python hive_fetcher.py --url 'https://hive.blog/completenoobs/@completenoobs/completenoobs-genesis'")
print("")
print(" # Fetch specific post by authorperm")
print(" python hive_fetcher.py --post '@completenoobs/completenoobs-genesis' --output genesis.xml")
if __name__ == "__main__":
main()
How to use script
usage: hive-fetch.py [-h] [--user USER] [--post POST] [--url URL] [--limit LIMIT] [--output OUTPUT] [--format {json,xml,wiki,txt}]
Fetch posts from Hive blockchain
options:
-h, --help show this help message and exit
--user USER Username to fetch posts from
--post POST Single post to fetch (format: @author/permlink or full URL)
--url URL Hive blog URL to fetch (e.g., https://hive.blog/@user/post)
--limit LIMIT Limit number of posts to fetch
--output OUTPUT Output file name (extension determines format)
--format {json,xml,wiki,txt}
Output format (json, xml, wiki, txt)
Examples:
# Fetch user posts as JSON
python hive_fetcher.py --user completenoobs --limit 10 --output posts.json
# Fetch user posts as XML
python hive_fetcher.py --user completenoobs --output posts.xml
# Fetch user posts as Wiki syntax
python hive_fetcher.py --user completenoobs --output posts.wiki
# Fetch specific post by URL
python hive_fetcher.py --url 'https://hive.blog/completenoobs/@completenoobs/completenoobs-genesis'
# Fetch specific post by authorperm
python hive_fetcher.py --post '@completenoobs/completenoobs-genesis' --output genesis.xml
Examples
- syntax_wiki example post: https://hive.blog/hive-138301/@completenoobs/find-hive-users-memo-key-and-send-encrypted-message-l7a
- syntax_mark example post: https://hive.blog/completenoobs/@completenoobs/n33bcoin-how-to-fork-bitcoin-for-learning-by-tinkering
To print text of post to terminal:
python3 hive-fetch.py --url https://hive.blog/hive-138301/@completenoobs/find-hive-users-memo-key-and-send-encrypted-message-l7a
To print syntax_wiki to file
python3 hive-fetch.py --url "https://hive.blog/hive-138301/@completenoobs/find-hive-users-memo-key-and-send-encrypted-message-l7a" --output test.wiki
To print syntax_markdown to syntax_wiki - BUGGY
python3 hive-fetch.py --url "https://hive.blog/completenoobs/@completenoobs/n33bcoin-how-to-fork-bitcoin-for-learning-by-tinkering" --output test2.wiki